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ABSTRACT 



Roads to Success (RTS) is a school and career planning program designed to be 
implemented for 45 minutes per week in grades 7 through 12. Researchers at Mathematica 
Policy Research used a random assignment design to estimate the impacts of receiving RTS in 
grades 7 and 8. More than half of the students in these schools were eligible for free or reduced- 
price lunches and the schools had few minority students. Using student survey data collected 
from more than 1,400 students, we found no evidence of statistically significant impacts of the 
RTS program on motivation to go to school to learn job skills or on learning and study habits at 
the end of grade 8. We did find a statistically significant impact at the .10 level suggesting that 
RTS reduced a composite measure of negative behaviors. However, exploratory analyses of 
subcomponents of this measure suggested mixed results for student behaviors. 

We conducted exploratory analyses of additional outcomes and found positive impacts of 
RTS on talking to school staff about career and school plans, confidence in knowing how to find 
out about what types of jobs are best, and confidence in knowing what is required to succeed in 
different careers. However, students in both the treatment and control group reported talking to 
their parents more than to school staff about most issues regarding school and career planning; 
more than 90 percent reported agreeing or strongly agreeing with the statement that they “will be 
able to overcome barriers that stand in the way of achieving my career goals.” Also, fewer than 
10 percent of the treatment and control group students reported career aspirations or expectations 
that seemed too high; fewer than one percent reported career aspirations that seemed too low. 
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I. INTRODUCTION AND BACKGROUND 



In early 2005, the Roads to Success (RTS) program was launched as an ongoing part of 
middle and high school programs to help forge connections between students’ school 
experiences and their aspirations for adulthood. In 2006, RTS partnered with Mathematica Policy 
Research (Mathematica) to conduct an eight-year evaluation of the intervention. This evaluation 
was designed to estimate impacts of RTS by using random assignment. Funding for the RTS 
program and study was cut severely in 2008 due to health problems of the funder; consequently, 
our final data collection occurred in spring 2009. This report describes the RTS intervention, 
research design, evaluation data, and analysis methods, and presents estimated impacts of 
receiving RTS in grades 7 and 8 on outcomes measured at the end of grade 8. 



A. SIGNIFICANCE 

While access to college has improved in the past 30 years, current educational practices do 
not adequately address the many steps needed to prepare students for success in high school, 
postsecondary education, and careers. Each year more than half a million young people drop out 
of high school; and this figure has been stable for most of the last 30 years (Heckman and 
Lafontaine 2007). This number is particularly troubling considering the high economic returns to 
education (Card 1999). In addition, occupations normally requiring a bachelor’s degree or more 
are expected to grow faster than others during the next decade (Dohm and Shniper 2007). 

Although federal and state policies are pushing high schools to focus on improving test 
scores, academic achievement is not the only obstacle to education and career success (Goble et 
al. 2006; DeLuca and Rosenbaum 2001; Roderick et al. 2006). One reason many youth continue 
to drop out of high school may be a lack of engagement in their education. Forty-seven percent 
of high school dropouts cite a lack of connection to school as the reason for leaving before they 
graduate (Bridgeland et al. 2006), and many students report that school is not relevant to their 
future careers (Steinberg 1996; Rosenbaum 2001). Although career plans and expectations are 
linked to students’ choices and school motivation, many schools do not consistently address 
them (Yazzie-Mintz 2007). Without developing clearer connections to their future careers, it may 
be difficult to increase student engagement in education. Many students either fail to understand 
the advantages afforded by a college degree or overestimate the barriers to attending college 
(Ikenberry and Hartle 1998). Unfortunately, overburdened high school guidance counselors often 
provide college information and advice in a single short meeting during a student’s senior year 
(McDonough 2005) and do not devote adequate time to the often complex college application 
process. Teachers are the de facto college resource when guidance counselors are unavailable 
and they are often unprepared to guide students through the process (Kirst et al. 2004). 

Lack of sufficient planning for their future education and career paths may have negative 
impacts on outcomes for many young adults. Inadequate education and career planning may 
result in students developing “unaligned ambitions” (Schneider and Stevenson 1999). Some 
students aim too low, seeking only “a few college classes” in fields where jobs require degrees, 
while others aim too high, planning degrees that will take more years of college than their 
budgeted amounts of time or money allow (Rosenbaum et al. 2006). Thus better career and 
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education planning that engages students and provides information about post-secondary options 
may be a vital component of K-12 education reform (Whiston et al. 1998; Oliver and Spokane 
1988; Lapan, Gysbers, and Sun 1997). 



B. THE RTS INTERVENTION 

RTS is an in-school guidance program designed to address (1) the lack of systematic 
guidance offered to students regarding their future and (2) the lack of engagement with school 
reported by many youth. Unlike the wide array of college access and guidance programs that are 
operating in high schools, RTS is a classroom-based program serving whole-grade cohorts at a 
low cost per student. Specially trained facilitators implement RTS. Key features of RTS include 
the following: 



• A Comprehensive and Consistent College and Career Planning Curriculum. A 
standardized curriculum covering career exploration, education planning, and 
education/workplace skills 

• Engaging Teaching Methods. Active classroom methods, technology, project-based 
learning activities, and student accountability 

• Weekly Delivery of Curriculum. Three-quarters of an hour per week for six school 
years (grades 7-12) 1 



The RTS curriculum is intended to improve postsecondary education and career outcomes 
by helping students (1) learn about career opportunities, (2) plan for their education 
appropriately, and (3) develop the skills needed to take full advantage of future educational and 
workplace opportunities (Figure 1). The early focus on career exploration is designed to increase 
student engagement by ensuring that youth see how their education relates to their future careers. 
The focus on education planning helps ensure that students complete the relevant high school 
coursework and related preparation activities in time to take full advantage of potential 
postsecondary and early career options. All three curriculum components are designed to help 
students succeed in critical transitions between grades, between schools, and when entering the 
workforce. 

RTS was created by a nonprofit organization (the RTS organization) that started designing 
the intervention in 2004. First, the organization examined the theoretical and empirical research 
base for a career and education planning program. It also met with numerous individuals and 
organizations to seek advice about specific program elements. Based on this work, the RTS 
organization staff created a scope and sequence of thematic units and lesson plans, which are 
summarized in the Grade Overviews in Appendix A of the design report (Chaplin et al. 2009). 
Outlines were developed for each lesson and then sent to professional curriculum writers, who 



1 Due to funding cuts in 2008, the treatment group students in this study received only two years of RTS 
(during grades 7 and 8). 
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developed complete lesson plans and related student handbook and portfolio materials. A sample 
lesson plan is available in Appendix B of Chaplin et al. 2009. The scope and sequence was 
revised several times up through the 2008-2009 school year, with input from content experts, an 
RTS advisory board, and school district partners. Quantitative and qualitative feedback has also 
been obtained from RTS facilitators after each lesson and from RTS students at the middle and 
end of each school year. 



FIGURE 1 

LOGIC MODEL FOR RTS PROGRAM 



The Intervention 

• Comprehensive and Consistent 
College and Career Planning 
Curriculum 

- Career exploration 

- Education planning 

- Education and workplace 
skills 

• Engaging Teaching Methods 

- Active classroom methods 

- Technology 

- Project-based learning 

- Student accountability 

• Weekly Delivery 

- 30 classes per year, each 45 
minutes long 

- All students in grades 7-12 
receive treatment 



Short-Term Outcomes 

• Increased engagement in 
school 

• Improved plans for 
education and career paths 

• Increased completion of 
college preparatory 
activities, including 
rigorous high school 
coursework and college 
and job applications 

• Successful transitions 
between schools and into 
the workforce, including 
high school completion and 
work during postsecondary 
education 



Long-Term Outcomes 

• Postsecondary 
education or 
training: 
enrollment, 
persistence, and 
completion 

• Career attainment 




Background Factors 



Pre-program student skills, School quality Parent and community 

knowledge, beliefs, and p eer e ff ccls characteristics 

behaviors 



The design of RTS is based in part on recommendations of the American School Counselors 
Association (ASCA) (Bowers and Hatch 2005). Although RTS is intended to help facilitate the 
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work of school counselors, it is not designed to replace them. Rather, by providing schools with 
a means of addressing the career and education planning needs of students, RTS is designed to 
help school counselors focus on the tasks that already consume much of their time — in 
particular, responding to the short-term needs of youth who require individual counseling, crisis 
management, referrals, and/or consultations (Bowers and Hatch 2005). Even though ASCA 
standards also call for counselors to develop classroom-based guidance programs, the reality is 
that most counselors, especially in low-income schools, do not have the time to do so (Lee and 
Ekstrom 1987). RTS is designed to address this reality. 



C. RESEARCH QUESTIONS AND STUDY DESIGN 

In fall 2006, Mathematica randomly assigned 25 middle and high schools to one of two 
treatment conditions. Thirteen of the schools were randomly assigned to deliver the RTS 
intervention to all grade 7 students during the 2007-2008 school year. These schools continued 
to deliver the RTS intervention to the same set of students in 2008-2009, when the students were 
in grade 8. The other 12 schools were randomly assigned to the control group and did not offer 
RTS to grade 7 students during the 2007-2008 school year or to grade 8 students in 2008-2009. 

Mathematica collected baseline test score data from the end of grade 6 (spring 2007), 
conducted a student baseline survey at the beginning of grade 7 (fall 2007), and conducted a 
student follow-up survey at the end of grade 8 (spring 2009). The current study is designed to 
answer two sets of research questions: 



1. What are the short-term impacts of the RTS program at the end of grade 8 for all 
students? 

2. How do the impacts vary by student and school characteristics? 

D. CONCEPTUAL BACKGROUND FOR THE STUDY 

Following is a detailed description of the key features of the RTS program, a discussion of 
its feasibility, and preliminary evidence of its efficacy. 



1. Comprehensive and Consistent College and Career Planning Curriculum 

The ASCA calls for the “systematic delivery” of a comprehensive guidance program 
(Bowers and Hatch 2005). In line with this recommendation, RTS lessons are part of a 
standardized, fully articulated six-year program. As noted earlier, the RTS curriculum follows a 
scope and sequence for each grade level (within a larger six-year scope) and includes detailed 
lesson plans (see Appendix B of Chaplin et al. 2009 for a sample lesson plan). It also uses 
student handbooks and portfolios and monthly parent newsletters that are aligned with program 
content (see Appendix C of Chaplin et al. 2009 for a sample parent newsletter). Although the 
balance of content differs by grade level, each year of the RTS program contains elements of 
career exploration, education planning, and education/workplace skills. To help ensure that RTS 
is implemented consistently at all sites, national RTS staff train and supervise the RTS 
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facilitators at each school to deliver the intervention to students. Staff from the national 
organization conduct biannual classroom observations of the facilitators as they deliver RTS 
lessons (see Appendix D of Chaplin et al. 2009 for a sample classroom observation protocol). 

In contrast, many interventions with similar goals may be implemented less consistently 
both within and across schools or program sites. For example, the federal Gaining Early 
Awareness and Readiness for Undergraduate Programs (GEAR UP) provides nearly 300 
different local grants, each supporting a different program mix developed from a general set of 
guidelines; the federal Talent Search program consists of approximately 360 grants with a 
similar framework (Humphrey et al. 2002). Evaluations of both programs have noted the 
difficulty of assessing their effectiveness due to the variable nature of “the program” (U.S. 
Department of Education 2002a, 2002b). 

Many programs similar to RTS target only a subset of students. Such programs that operate 
as more of a “pullout” for a subset of students (for example, Talent Search) can suffer from 
highly variable student participation, particularly over successive years of enrollment in a 
program. Similarly, programs that are offered after school, during the summer, or as electives, 
also suffer from highly variable participation and limit which students may participate (U.S. 
Department of Education 2002b). In contrast, RTS is intended as a mandatory program for an 
entire cohort, serving all students in a school or given grade during the school day as a regular 
part of each student’s schedule. This treatment regimen helps ensure that the program is 
delivered in a consistent manner and has the potential to create a peer culture focused on 
motivation, planning, and educational attainment (Hossler et al. 1999; McDonough 2005). 
Following is a detailed description of the curriculum elements of the RTS intervention. 



a. Career Exploration 

Career exploration is a key early component of the RTS intervention because of the need to 
motivate students to plan for their education and develop appropriate education and workplace 
skills. Research suggests that career planning is positively associated with student engagement in 
both middle and high school (Kenny et al. 2006; Lapan et al. 2001; Lapan 2004). Hossler et al. 
(1999) argue that, in order to maximize their chances of postsecondary and career success, 
students should develop career and educational aspirations in grades 7 to 9. Meta-analyses 
suggest that career guidance activities may have the largest positive effects in junior high or 
middle school (Oliver and Spokane 1988; Whiston et al. 1998). 

Consistent with these findings, the RTS program contains a significant yearly focus on 
career exploration and planning starting in grade 7, when students are starting to decide whether 
to go to college. The goal is to increase student motivation through the development of career 
aspirations before students enter high school. The content for grades 7 through 9 includes interest 
inventories; a grade 7 student-led career fair; lessons on identifying careers of interest; the 
education required to enter those careers; and the benefits of a college education. In later grades, 
additional career exploration activities are presented, including a grade 11 job shadowing project. 
Appendix A of Chaplin et al. 2009 includes an overview of the RTS curriculum by grade. 
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b. Education Planning 

Students who decide to go to college must have sufficient information about how to search 
successfully for appropriate postsecondary alternatives and how to choose among them (Hossler 
and Gallagher 1987). Many students and their families lack this information (Cabrera and La 
Nasa 2000). Some overestimate the cost of college and are unaware of available financial 
assistance (U.S. Department of Education 2003). The RTS program provides students and their 
families with this information. Although good information is important, that alone is not enough. 
Many students can state their career aspirations, but do not understand what makes those 
aspirations desirable, what it would be like to have those jobs, or what actions they must take to 
make their goals a reality. Bridging this gap in understanding may be difficult for guidance 
counselors in a typical 15-minute session with a student. Consequently, RTS incorporates 
planning skills into its curriculum. 

The importance of planning is suggested by the fact that adolescents with coherent, aligned 
plans demonstrate greater motivation and effort and have an increased capacity to draw on 
resources (Schneider and Stevenson 1999). More generally, experts suggest that helping students 
plan for their education and careers appears to be important for high school completion and later 
outcomes (Dynarski et al. 2008; Woloszyk 1996; Hayward and Tallmadge 1995; Bragg 1997; 
Bauer and Michael 1993). 

The RTS six-year sequence of activities for students and monthly newsletters for parents are 
designed to ensure that students and families have sufficient information and planning skills to 
make informed educational choices. Beginning in grade 7, students learn about postsecondary 
options; in grade 10, students take part in a college visit and prepare for the PSAT during class 
time. RTS activities in grades 11 and 12 focus class time on the specific activities in which 
students must engage to attend college — including preparing for the SAT/ACT, filling out 
college and financial aid applications, and engaging in scholarship search and postsecondary 
budgeting activities. They also participate in job shadowing in grade 11 and in informational 
interviewing in grade 12. RTS students develop education plans early in the process, revisit the 
plans each year, and receive advice to help them understand their plans as well as encouragement 
to make sure that their plans are aligned with career goals, personal interests, values, and 
academic preparation. 

Students with a low socioeconomic background are less likely than other students to have 
access to education planning and other related guidance activities needed for a successful 
secondary and postsecondary experience. Furthermore, students who lack access to guidance 
counseling are more likely to be placed in non-academic curricular tracks and take fewer 
mathematics courses (Lee and Ekstrom 1987). Consequently, all RTS students participate in the 
planning and researching of education options, helping ensure that students receive the 
counseling and information they need to make informed decisions about the courses they take in 
high school and about their post-high school careers. 



c. Education and Workplace Skills 

Employers readily acknowledge that America’s high school graduates lack many non- 
academic skills needed for success in the workplace (Olson 2007) and the competitive global 



6 




economy (Hamilton and Hamilton 1999; Orfield 1997; Conley 2007). Similarly, educators 
believe that many college students lack key skills necessary for postsecondary success 
(Rosenbaum et al. 2006). Indeed, both sets of skills, many of which overlap, receive scant 
attention in most secondary schools in the United States compared to other nations (Stevenson 
and Stigler 1992). To address these issues, RTS includes components designed to teach a number 
of non-academic skills, including information gathering, planning, group work, presentation 
methods, and financial management. 

The RTS middle school program includes lessons on short-term planning, study skills, note- 
taking, managing stress, and budgeting. The grade 7 career fair introduces students to the 
rudiments of research and public speaking, as does a grade 8 project-based learning unit on 
improving one’s community. Students learn how to work in groups effectively, reflect on 
projects, and complete tasks in long-term projects. 

The high school program covers information-gathering skills through lessons on researching 
postsecondary and career opportunities. Self-presentation skills are enhanced through lessons on 
interviewing, networking, and preparing a resume. Finally, financial literacy activities include 
planning a postsecondary budget and learning about credit, insurance, and consumer rights. 

Both the middle and high school components of RTS cover test-taking lessons, with a focus 
on the PSAT and SAT/ACT. Test-taking skills taught in a vacuum may yield little benefit 
(Allensworth et al. 2008); therefore, RTS embeds this work into a broader focus on the 
development of general reasoning skills and career and education planning to help ensure that 
students are engaged when developing these skills. The lessons cover samples of tests required in 
non-college contexts, such as tests for employment as a Federal Express courier or admission to 
an electrician’s apprenticeship program. The lessons are designed to inform students of the 
existence and purpose of these tests, to fa mi liarize them with the types of questions they will 
encounter, connect them with other resources to help them prepare for future tests, walk them 
through the registration process for the SAT or ACT, and to underscore more generally the need 
for good mathematics and reading skills for a variety of postsecondary paths. 



2. Engaging Teaching Methods 

To ensure that students leam the topics covered in the curriculum, RTS uses methods 
designed to maximize student engagement, including the use of active teaching methods, 
technology, annual project-based learning activities, and student accountability. 



a. Active Teaching Methods 

Research suggests that students learn and retain more when they are active participants and 
can relate to the content that is taught (Akey 2006). Career exploration appears to encourage 
student engagement (Kenny et al. 2006). The evidence also suggests that students benefit when 
teachers use a wide range of instructional strategies, such as group activities, long-term projects, 
hands-on activities, lessons that draw from student interests, and cooperative learning, to engage 
divergent student interest in learning (Garcia-Reid et al. 2005; Akey 2006; Heller et al. 2003; 
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Wynne 1995). For these reasons RTS instructional methods are designed to require a minimum 
amount of lecturing. 



b. Technology 

RTS uses educational technologies to encourage student engagement primarily through the 
use of web-based career exploration and college research programs such as Career Cruising 
(www.careercruising.com). Some evidence on career planning interventions suggests that 
combining individual counseling with computer applications is more effective than individual 
counseling alone (Whiston et al. 1998). 



c. Project-Based Learning 

Experiential education has demonstrated the ability to raise student engagement in learning 
(Akey 2006; Heller et al. 2003). In RTS, students participate in many types of project-based 
learning as discussed earlier, including a grade 7 career fair project, a grade 8 community 
improvement project, and a grade 9 workplace- simulation project. In grade 10, RTS students 
make a college visit; in grade 11, they participate in a job shadowing experience; and in grade 
12, students apply for colleges and/or jobs. 



d. Student Accountability 

Being held accountable for their performance in school activities can also encourage 
students to be more engaged (Kozik et al. 2005). RTS students receive pass/fail grades that are 
reported to parents either by the school or RTS. In some cases, performance in RTS counts 
toward student grades in non-RTS classes. 



3. Weekly Delivery of Curriculum 

RTS was designed to be a weekly long-term intervention. It includes approximately 30 
lessons per year, each 45 minutes long, beginning in grade 7 and continuing through grade 12. 
By contrast, most career-related interventions are short term (Oliver and Spokane 1988), and 
many college preparatory interventions (for example, College Summit) are designed for a narrow 
grade span. This creates the risk of getting to students too late to help many of them and not 
persisting long enough to make a difference (Whiston et al. 1998). 

Due to funding cuts, this study estimates impacts of an abbreviated version of the 
intervention that includes only two years of exposure (grades 7 and 8). Thus this study does not 
enable us to estimate the impact of the RTS intervention when fully implemented in grades 7 
through 12. 
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4. Feasibility 



RTS has been designed to be implemented without undue effort on a school’s part. Time 
demands on a school’s academic schedule are minimal — about two percent of a student’s school 
time over the course of a year — but the cumulative program dosage of the full six-year program 
is sizable. The format reflects the reality that the demands of state standards and federal law 
(including No Child Left Behind requirements) prevent a greater allocation of time to classroom- 
based guidance activities. 

Direct program costs — including facilitator compensation, training and support, student 
materials, classroom supplies, and technology — average about $300 per student per year, or 
about three percent of the average 2006 public school spending rate of more than $9,000 per 
student (U.S. Census Bureau 2008). As important, RTS meets many of the requirements of 
existing funding streams such as the federal GEAR UP program and thus may be incorporated 
into current and new GEAR UP grants as a standardized component of such programs. 

There is significant evidence of RTS feasibility in authentic education delivery settings. 
Since 2005, RTS has been implemented in public secondary schools in three states — New York, 
Pennsylvania, and West Virginia. During the 2008-2009 school year, it was implemented in a 
mix of 30 urban and rural low-income schools (see Appendix E of Chaplin et al. 2009 for a 
complete list) and served more than 4,000 students (an increase from 3,500 students in 23 
schools during the 2007-2008 school year). 

In addition to collecting evidence reflecting the commitment of schools and districts to the 
program, the RTS organization collects evidence of lesson-specific implementation. These data 
are collected weekly by RTS facilitators using implementation forms listing which classes were 
taught and the percentages of each lesson delivered to each class (see the Facilitator 
Implementation Form in Appendix F of Chaplin et al. 2009 for the form used during the 2008- 
2009 school year). Using these data we estimate that the RTS facilitators reported covering 88 
percent of the grade 7 material during the 2007-2008 school year and 93 percent of the grade 8 
material during the 2008-2009 school year in the treatment schools covered by our study. Using 
data from the 2007-2008 school year, the RTS staff reviewed lessons with low implementation 
rates for comparison with written comments provided in journals by the facilitators and then 
revised and/or compressed several lessons. 



5. Potential Efficacy 

Prior to the current study, the RTS organization administered anonymous end-of-year 
student feedback forms to an earlier cohort of RTS students to help obtain early evidence on the 
potential efficacy of the RTS intervention. These feedback forms included several questions 
taken from the High School Survey of Student Engagement (HSSSE). The HSSSE was 
administered in the same year (2006-2007) to a diverse sample of students spread across 26 
states (Yazzie-Mintz 2007). The HSSSE students were asked how much their overall school 
experience had helped them leam work skills, work well with others, solve real-world problems, 
develop career goals, and understand themselves; the RTS feedback form asked the same 
questions with respect to how much RTS had helped RTS students in the same areas. For each 
question, student choices included “Very Much,” “Some,” “A Little,” and “Not at All.” Table 1.1 
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presents the percentages of students responding “Very Much” to each question. Assuming simple 
random sampling, all of the differences between outcomes for the RTS and HSSSE students are 
statistically significant at the 0.01 level. 2 The results suggest that RTS participants believed that 
the RTS program helped them learn these types of skills more than students in the HSSSE study 
felt that their school experiences helped them. 

TABLE 1. 1 

STUDENT REPORTS ON WHETHER RTS/SCHOOL EXPERIENCES 
HELP LEARNING. BY TOPIC 



Skill 


Percentage Answering 


“Very Much” 


RTS students 


HSSSE students 


Work skills 


38% 


23% 


Working well with others 


45% 


29% 


Solving real-world problems 


38% 


20% 


Developing career goals 


53% 


23% 


Understanding yourself 


56% 


25% 



Source: The HSSSE survey for the HSSSE students and data collected by RTS staff on the RTS students. 

Note: There are 1,100 RTS students from 7 schools and 81,499 HSSSE students from 110 schools covered in this 

table. All data are for the 2006-2007 school year, prior to the beginning of the current study. All 
differences between the RTS and HSSSE student outcomes are statistically significant at the 0.01 level 
using two-tailed tests and assuming simple random sampling. They all have /-statistics greater than 10. 



E. ORGANIZATION AND CONTENT OF THIS REPORT 

The rest of this report presents the methods, data, and impact findings. Chapter II describes 
the study design, methods, and random assignment of schools. Chapter III discusses the data 
collection process, including informed consent of both parents and students, and presents 
response rates for each student survey. Chapter IV presents the research questions and estimation 
approach. In Chapter V, we present estimates of the impact of RTS on student motivation, 
learning and study habits, and attendance and negative behaviors; these are followed by the 



2 All tests of statistical significance are based on two-tailed tests. Unless stated otherwise, findings that are 
reported as “not statistically significant” have /^-values > 0.10, the highest value typically used to define statistical 
significance and findings that are reported as “statistically significant” have /^-values <0.01, a relatively low 
(stringent) cut off for statistical significance. Thus we are erring on the side of caution in both directions. For other 
findings we specify an upper bound for the significance level. 

3 For the comparisons in Table 1.1, we did not have access to the student-level data, so we could not correct for 
clustering by school, student background characteristics, or the complexity of the HSSSE survey design. However, 
the /-statistics we did calculate (assuming simple random sampling) were all greater than 10. 
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estimated impacts on school engagement in school, career exploration behavior, and career 
exploration efficacy. In Chapter VI, we present our conclusions. 
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II. STUDY DESIGN AND METHODS 



A. SELECTION OF SCHOOLS 

RTS recruited the 25 middle and high schools in this study based on their regional proximity 
to each other, their willingness to participate in the study, and the apparent availability of data 
that would facilitate the proposed analyses. The study was originally designed to be an eight-year 
evaluation of the intervention and involved following the students through high school. Thus, the 
study schools were also selected so that (1) most students in each of the selected middle/junior 
high schools attend only one corresponding high school and (2) each high school in the study 
receives students from only one middle/junior high school in the study. This method was used to 
help ensure that treatment and control group members from the same cohort in different junior 
high/middle schools would not end up in the same high school. As mentioned earlier, funding for 
the RTS program and study was cut severely in 2008 due to health problems of the funder. 
Consequently, the study ended following the 2008-2009 school year, and the program also 
ended in most of the study schools at that time. 

The districts that participated in the program are low-income, rural districts with lower 
income and education levels than the averages for their state. The 25 schools that were randomly 
assigned came from three geographic areas — 9 schools in New York; 6 in Wayne County, West 
Virginia; and 10 in other counties in West Virginia. All of the schools are in rural counties, are 
more than 94 percent white, and have attendance rates of at least 89 percent. 



B. RANDOM ASSIGNMENT OF SCHOOLS TO TREATMENT CONDITION 

In fall 2006, Mathematica randomly assigned the 25 schools to one of two treatment 
conditions. Thirteen of the schools were randomly assigned to deliver the RTS intervention to all 
grade 7 students during the 2007-2008 school year. These schools continued to deliver the RTS 
intervention to the same set of students in 2008-2009, when the students were in grade 8. The 
other 12 schools were randomly assigned to the control group and did not offer RTS to grade 7 
students during the 2007-2008 school year or to grade 8 students in 2008-2009. 

To ensure that the sample was balanced and to increase precision, we grouped the 25 
schools into blocks of two or three schools each before random assignment. These blocks were 
chosen so that schools in each block came from one of three geographic areas — Western New 
York and two parts of rural West Virginia. Within these areas, Mathematica matched schools 
based on the fraction of students eligible for free or reduced-price lunch and school average test 
scores. 

To combine the school-level data on free lunch eligibility and test scores used to match 
schools, we first calculated an average test score variable equal to the average of the z-scores for 
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each relevant test score variable we had in the data. 4 The test score variables covered the 
fractions of students proficient in mathematics, science, and English (separately). We then took 
an average of a z- score transformation of the average test score variable and a negative z- score 
transformation of the free lunch variable. We used the negative value of the free lunch variable 
because higher free lunch rates are generally associated with lower test scores. 

The blocking resulted in 12 blocks of schools — 11 blocks with two schools each and one 
block with three schools. 5 One school was chosen randomly within each block to deliver the 
RTS intervention to grade 7 students starting in fall 2007, except in the block with three schools, 
where two of the three schools were chosen. The remaining schools were allowed to have a later 
cohort of grade 7 students participate in RTS starting in the 2008-2009 school year. The students 
in the later cohort are not included in the current report. 

C. SCHOOL-LEVEL TREATMENT-CONTROL BALANCE AT BASELINE 

In order to test the balance between the treatment and control groups, we compared baseline 
characteristics of the two sets of schools based on school-level data that we collected from state 
and district websites. 

Table II. 1 presents means of these variables for the treatment and control schools. None of 
these differences is statistically significant, and neither is a joint significance test of the 
differences for the free lunch and test score variables combined. The differences between the two 
group means are also generally small in a substantive sense. For example, there is only a five 
percentage point difference in the free and reduced-price lunch eligibility rate, around 0.1 of a 
standard deviation. That is similar to the year-to-year changes observed in many of these schools. 
The differences in the remaining variables are even smaller in terms of standard deviation units, 
as shown in the last column of Table ILL 



D. STUDENT-LEVEL TREATMENT-CONTROL BALANCE AT BASELINE 

The baseline characteristics reported in Table II. 1 cover all of the schools in the original 
sample and all students at those schools. After random assignment, two schools dropped out of 
the study and many students in the remaining schools did not respond to the survey (see Chapter 
III for response rates). To deal with non response at the student level, we used weights, designed 
so that our results provide unbiased estimates of the impacts of RTS for responding students in 
responding schools. 6 A detailed description of our weighting procedure is provided below in the 
section on impact estimation methods. 



4 The z-scores were created by subtracting the mean within the geographic area and then dividing by the 
standard deviation at the school level for that area; the test scores were analyzed separately for each grade and 
subject. 

5 Only 23 of the original sample of 25 schools are used when estimating impacts at the end of grade 8 because 
one matched pair of schools dropped out of the study. 

6 The two non-responding schools came from the same matched block. Thus, our estimates do not cover that 
block. 
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TABLE II. 1 



COMPARING SCHOOL-LEVEL MEANS 



Variable 


Treatment 

Schools 


Control 

Schools 


Difference 


Standard 

Deviation 


Difference/S tandard 
Deviation 


Free and reduced-price lunch 


0.51 


0.56 


0.05 


0.50 


0.10 


Mathematics 


0.66 


0.67 


0.01 


0.47 


0.02 


English 


0.67 


0.70 


0.03 


0.46 


0.06 


Science 


0.83 


0.82 


-0.01 


0.38 


-0.03 


Attendance 


0.95 


0.96 


0.01 


0.21 


0.05 



Source: School-level data collected by Mathematica and RTS staff from state and district websites. 

Notes: Standard deviation is at student level. 

No differences are statistically significant at the 10 percent level. 

Mathematics/English/Science = fraction of students tested proficient in subject. 

West Virginia tests combined grade 7 results from the 2004-2005 school year with grade 6 results 
from the 2003-2004 school year’s West Virginia Educational Standards Test (Westest). 

New York tests based on grade 8 results from the 2004-2005 school year intermediate-level tests. 
Attendance = attendance rate at school in 2005-2006. 



Table II. 2 presents unweighted averages of baseline characteristics from the fall 2007 
student survey and from grade 6 baseline test scores for the students in our study. The 
differences are not statistically significant at the 0.05 level, either individually or jointly. 
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TABLE II.2 



COMPARING STUDENT-LEVEL MEANS 



Variable 


Treatment 

Students 


Control 

Students 


Difference 


Sample 

Size 


Female 


50.4% 


52.4% 


-2.4 


1,590 


Race/Ethnicity 


White 


91.5% 


90.4% 


1.1 


1,590 


Black or African American 


1.3% 


1.4% 


-0.1 


1,590 


Hispanic or Latino/Latina 


0.4% 


0.8% 


-0.4 


1,590 


Asian 


0.4% 


0.4% 


0.0 


1,590 


American Indian or Alaska Native 


0.1% 


0.1% 


0.0 


1,590 


Other/mixed/unknown 


6.1% 


6.9% 


-0.8 


1,590 


Mother’s Highest Level of Education 


At least high school diploma 


68.4% 


64.8% 


3.6 


1,590 


At least bachelor's degree 


23.9% 


22.8% 


1.1 


1,590 


Father’s Highest Level of Education 


At least high school diploma 


61.5% 


57.1% 


4.4 


1,590 


At least bachelor's degree 


15.6% 


14.6% 


1.0 


1,590 


English Spoken at Home 


98.6% 


98.8% 


-0.2 


1,590 


Computer Access at Home 


82.3% 


80.4% 


1.9 


1,590 


New York Test Scores Means (SD) 


Math proficiency 


2.8 (0.81) 


2.9 (0.84) 


-0.1 


232 


Reading proficiency 


2.8 (0.63) 


2.7 (0.62) 


0.1 


232 


Math scale score 


663.4 (39.32) 


673.4 (42.08) 


-10.1 


232 


Reading scale score 


661.5 (29.28) 


660.6(31.16) 


0.9 


232 


West Virginia Test Scores Means (SD) 


Math proficiency 


3.0 (0.88) 


3.1 (0.91) 


-0.1 


1,358 


Reading proficiency 


3.3 (0.82) 


3.3 (0.92) 


0.0 


1,358 


Math scale score 


675.7 (34.17) 


678.2 (34.47) 


-2.5 


1,358 


Reading scale score 


670.8 (26.37) 


670.3 (30.60) 


0.5 


1,358 



Source: Mathematica student surveys administered to all study students in fall 2007 and spring 2009; if data were 
missing for the baseline survey, we used the follow-up survey data from 2009 covering the same 
information. Baseline test score data are from grade 6 standardized math and English/language arts tests 
from spring 2007. The sample includes only students with complete data on all of the variables in the table. 

Notes: These results are unweighted. None of the differences are statistically significant at the 0.05 level using a 

two-tailed test. SD = standard deviation. 



16 




III. DATA 



A. DATA COLLECTION METHODS 

In accordance with the conceptual framework presented in Chapter II, Mathematica 
collected detailed data on students’ learning and study habits, attitudes toward school, career 
efficacy, and important background characteristics that may have influenced student outcomes. 
We administered a baseline student survey in fall 2007, at which time we also requested passive 
parental consent and student assent to participate in the study. In spring 2008, we collected grade 
6 baseline test score data from spring 2007. In spring 2009, we administered a follow-up student 
survey and collected school-level attendance data for the 2008-2009 school year. We requested 
student rosters to track student mobility during both years of the study. Table III.l presents a 
schedule of the data collection activities during each year of the study. A brief description of 
each data collection activity is provided below. 

TABLE III.l 

PROJECT DATA COLLECTION SCHEDULE 





Calendar Year 


2006-2007 


2007-2008 


2008-2009 


Grade Level of Students 


6 


7 


8 


Data Collection Activity 


Spring 


Fall 


Spring 


Fall 


Spring 


Student test score data (collected in spring 2008) 


X 










Student rosters and mobility data 




X 


X 


X 


X 


Baseline student survey 




X 








Follow-up student survey 










X 


Follow-up attendance data 










X 



Note: The baseline student survey was administered a few weeks after the RTS intervention had begun in some 

tteatment schools; the follow-up student survey was administered a few weeks before the conclusion of the 
grade 8 RTS intervention. 



Our informed consent procedure was based on a process approved by the Public/Private 
Ventures Internal Review Board. In fall 2007, all schools sent informational letters to parents 
describing the study and the types of data to be collected. Parents in all but three schools were 
given the option to contact Mathematica to request that their child not participate in the study. 
We did not receive any such requests from parents in fall 2007, so all students in these schools 
were eligible to participate in the study. The remaining three schools in the study required active 
consent during both the 2007-2008 and 2008-2009 school years: parents had to sign and return a 
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consent form to their child’s school in order for their child to participate during that year of the 

7 

study. Not all parents signed and returned the form. 

Students who received passive or active parental consent were asked to read and sign a 
student assent form prior to completing the baseline and follow-up surveys. Students were 
allowed to decline participation. Students who did not receive parental consent were not asked to 
complete the student assent form or participate in the student survey. 



B. STUDENT ADMINISTRATIVE DATA 
1. Student Rosters and Mobility Data 

At the beginning of each school semester (fall 2007, spring 2008, fall 2008, and spring 
2009), Mathematica requested student rosters from each school. Some schools were able to 
export electronic files and others sent hard-copy student rosters. In addition to student names and 
identification numbers, some school rosters also included each student’s gender and birth date. 
Schools were asked to provide the names of previous and/or subsequent schools if study students 
moved in or out of study schools. Mathematica used this student roster information to track 
student mobility in and out of study schools and districts during both years of the study. 



2. Student Test Score Data 

In spring 2008, Mathematica collected baseline test score data based on standardized math 
and English/language arts tests that study students had taken in spring 2007, while they were in 
grade 6. Mathematica worked with schools and districts to determine the most appropriate 
method for collecting these data. Some schools and districts were able to export electronic files 
containing test score data; others sent hard-copy test score reports to Mathematica. Mathematica 
asked schools to send scale scores and performance levels from both math and English/language 
arts state standardized tests, but some schools sent only one. 

Mathematica obtained an 81 percent response rate for the grade 6 test scores among the 
sample of students enrolled in the original set of randomized schools in fall 2007. This response 
rate includes the school that dropped out of the study and three schools that required active 
parental consent. 



Because the baseline test scores are from two different states, the scores must be 
standardized so that they fit comparable scales. We transformed the test scores by subtracting 
from each student’s score the sample mean for that test, subject, grade, and year and dividing by 



7 Two of the schools that required active consent were in the same block and had student response rates of 51.2 
and 38.5 percent for the spring 2009 follow-up survey. The third school that required active consent was in a 
separate block and had a student response rate of 53 percent. This school was paired with a school that had a student 
response rate of 85 percent. The remaining schools, which required passive parental consent, had response rates 
ranging from 58 to 91 percent. 
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the sample standard deviation for that test, subject, grade, and year. 8 This yields a standardized 
“z-score” that equates each student to the average student in the sample and that is comparable 
across all the students in the sample. 



3. Follow-Up Student Attendance Data 

In spring 2009, Mathematica contacted schools to collect school-level student attendance 
information. Eighteen schools provided information that could be used to calculate the average 
number of days that grade 8 students were absent during the 2008-2009 school year. These data 
were used to help validate the attendance question on our student survey instrument. We found a 
positive correlation of 0.28 between the school-level reported attendance rates and the average of 
the self-reported survey responses. Results by gender were available for three schools; for these 
schools, the results by gender did not match as well. Schools reported that females had higher 
attendance levels than males, whereas in the self-reported data, the opposite was true. 
Nevertheless, the differences between the female/male differentials on the two sets of measures 
were not statistically significant. 

School- level attendance data may have differed from student-level data for several reasons: 
(1) not all students responded to the student survey, but all students were included in school-level 
attendance data; (2) the response scale for the student survey item included only four categorical 
response categories whereas the school measure was open-ended; and (3) schools may have used 
various methods for calculating absences (for instance, some schools may have categorized 
missing one period as an absence, whereas other schools might require a student to miss the 
whole day before categorizing it as an absence). 



C. STUDENT SURVEYS 
1. Baseline Student Survey 

Mathematica developed and oversaw administration of a baseline student survey in fall 
2007, when students were in grade 7. At the beginning of the 2007-2008 school year, each 
school sent student rosters to Mathematica. Mathematica prepared and labeled student surveys 
for each treatment and control student. Survey packets were then disseminated to the relevant 
teachers using the class rosters provided by the schools. Packets contained a survey, student 
assent form, and envelope for each eligible student; an instruction sheet for the teacher; and 
postage-paid return materials. 

The survey was administered in RTS classes in treatment schools and in homeroom 
classrooms in control schools. Teachers first asked students to read and sign the student assent 
form. Teachers then asked the students who had provided assent to peel off and throw away their 
name from the outer label of the survey, leaving only the student’s study ID number. Students 
completed the 30-minute paper-and-pencil survey during class and sealed the completed 



8 Students for whom scaled scores were imputed are included in the sample mean for calculating the z-scores 
but not in the sample standard deviation. 
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instrument in the envelope. Teachers then mailed the surveys back to Mathematica. When 
needed, Mathematica staff sent reminder emails to RTS facilitators or other school liaisons to 
prompt teachers to return the surveys. 

One of the schools originally assigned to the treatment group did not participate in the 
administration of the student survey in fall 2007. Also, as mentioned above, three study schools 
required active parental consent (parents had to sign and return a consent form to their child’s 
school in order for their child to participate in the survey). Both of these occurrences decreased 
the response rate for the survey. Overall, Mathematica obtained a 79 percent response rate on the 
baseline survey among the sample of students from the original set of randomized schools (Table 
III.2). The difference in response rates between the treatment and control groups was not 
statistically significant. 



TABLE III.2 

STUDENT-LEVEL RESPONSE RATES FOR BASELINE SURVEY, FALL 2007 





Student 

Survey 

Complete 


No 

Student 

Assent 


No 

Parental 

Consent 


Student 

Absent 


No Survey 
Booklet 
Returned 


No School 
Participation 


Total 


Treatment Group 


N 


891 


28 


30 


88 


16 


96 


1,149 


Percentage 


78% 


2% 


3% 


8% 


1% 


8% 


100% 


Control Group 


N 


804 


5 


84 


102 


12 


0 


1,007 


Percentage 


80% 


0% 


8% 


10% 


1% 


0% 


100% 


Total 


N 


1,695 


33 


114 


190 


28 


96 


2,156 


Percentage 


79% 


2% 


5% 


9% 


1% 


4% 


100% 



Source: Mathematica student surveys administered in fall 2007. 

Note: The response rates for the baseline survey are based on the sample of grade 7 students in the original set of 

randomized schools in fall 2007. 



2. Follow-Up Student Survey 

Mathematica administered a follow-up student survey in spring 2009, when students were 
finishing grade 8 (see follow-up survey instrument in Appendix A), using the same procedures 
that were used for the baseline student survey. One of the schools originally assigned to the 
treatment group and one of the schools originally assigned to the control group did not 
participate in the administration of the student survey in spring 2009. These two schools were 
paired in the same block during our original random assignment, so this block of schools is 
dropped from the impact analyses. Also, as mentioned earlier, three study schools required active 
parental consent in order for their child to participate in the survey. These occurrences decreased 
the response rate for the survey. Overall, we obtained a 67 percent response rate on the follow-up 
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survey among the sample of students that was enrolled in the original set of randomized schools 
in fall 2007 (Table III.3). 9 Sixty-two percent of the original sample of students responded to both 
the baseline and follow-up surveys. Our final analysis sample consists of 1,305 students (60.5 
percent of the original baseline sample). Some students are omitted from our impact analysis 
because of missing data on specific survey items or because they did not respond to at least one 
of the two surveys. 



TABLE III. 3 

STUDENT-LEVEL RESPONSE RATES FOR FOLLOW-UP SURVEY, SPRING 2009 





Student 

Survey 

Complete 


No 

Student 

Assent 


No 

Parental 

Consent 


Student 

Absent 


No 

Survey 

Booklet 

Returned 


No School 
Participation 


Student 
Moved 
Out of 
Study 


Total 


Treatment Group 


N 


800 


5 


16 


104 


11 


96 


117 


1,149 


Percentage 


70% 


0% 


1% 


9% 


1% 


8% 


10% 


100% 


Control group 


N 


651 


7 


84 


114 


9 


52 


90 


1,007 


Percentage 


65% 


1% 


8% 


11% 


1% 


5% 


9% 


100% 


Total 


N 


1,451 


12 


100 


218 


20 


148 


207 


2,156 


Percentage 


67% 


1% 


5% 


10% 


1% 


7% 


10% 


100% 



Source: Mathematica student surveys administered in spring 2009. 

Note: The response rates for the follow-up survey are based on the sample of grade 7 students in the original set 

of randomized schools in fall 2007 (the same denominators as in Table III.2). 



9 



The treatment-control difference in response rates to the follow-up survey was not statistically significant. 
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IV. RESEARCH QUESTIONS AND IMPACT ESTIMATION 



Impacts of the RTS intervention were estimated by comparing outcomes for the treatment 
and control groups, after the treatment schools had received two years of the RTS intervention. 
The research design ensures that the estimates are unbiased, but we control for background 
factors to add precision to the estimates by using appropriate multivariate regression methods. 
We adjust standard errors for clustering of students within schools. 

We used student survey data and school administrative data to estimate program impacts. 
The survey data enable us to estimate impacts on a number of outcomes not available in 
administrative data, such as students’ habits, attitudes, and plans and any early activities students 
take part in as they prepare for postsecondary education and careers. Survey data on 
demographic characteristics (age, gender, race/ethnicity, and language spoken at home) are used 
to show how estimated impacts of RTS vary with these characteristics. The administrative data 
collected from schools (or school districts when possible) enable us to ascertain how impacts 
vary with characteristics such as pre-RTS grade 6 test scores. 

Our impact analysis focuses on two types of outcomes — a confirmatory set and an 
exploratory set. The confirmatory outcomes are used to test how well our pre-specified 
hypotheses (laid out in Chaplin et al. 2009) are supported by the data. Our main findings are 
based on these results. The exploratory outcomes are used to test additional hypotheses that 
might be the basis for more rigorous testing in later studies. We made this distinction, based on 
recommendations by Schochet (2008), in order to reduce the likelihood of reporting both false 
positives and false negatives in our results. Our confirmatory and exploratory sets of outcomes 
for grade 8 are based on the design of the RTS intervention and the degree to which similar 
variables have been shown to be associated with later measures of success such as postsecondary 
educational and career outcomes. The confirmatory and exploratory outcomes are described 
below. 

As noted earlier, although RTS was originally designed as a six-year intervention, this study 
ended after students in the treatment group had received only two years of RTS. In Appendix C 
we describe additional outcomes that we were planning to measure at the end of the six-year 
program. 



A. CONFIRMATORY OUTCOMES 

The three confirmatory outcomes of interest for grade 8 are (1) motivation to go to school to 
learn job skills (Question Alb in the follow-up survey in Appendix A), (2) learning and study 
habits/preparation (Question B1 in the follow-up survey), and (3) school attendance and negative 
behaviors (Question B3 in the follow-up survey). 10 Research by Rosenbaum (2001) suggests that 



10 Questions B1 and B3 contain multiple sub-questions. We took the average response across the sub- 
questions. 
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students’ motivation to go to school (that is, future relevance) strongly predicts their efforts in 
school and that poor attendance and discipline problems have significant negative effects on 
future earnings 10 years after high school. 

We estimated the impacts of assignment to RTS on these three main outcomes separately. In 
addition, we tested for the joint significance of these coefficient estimates, one of the options 
recommended by Schochet (2008) when doing confirmatory analyses for outcomes that differ in 
substantively important ways. Conducting a joint significance test reduces the likelihood of 
having a false positive finding, compared to not conducting such a test. In addition it avoids 
complications associated with choosing how to combine disparate outcomes (another method of 
avoiding false positives). 



B. EXPLORATORY OUTCOMES 

For our exploratory analyses, we estimated impacts of RTS on a number of other outcomes 
at the end of grade 8, including (1) the five sub-questions used to create the confirmatory 
outcome, school attendance and negative behaviors; (2) career exploration behavior with parents 
(Question A6 in the follow-up student survey); (3) career exploration behavior with 
teachers/school staff (Question A7 in the follow-up student survey); 11 (4) school engagement 
(Questions Ala and A2 in the follow-up survey); (5) importance of grades (Question A2 in the 
follow-up survey); (6) career exploration efficacy (Question Cll in the follow-up survey); and 
(6) desired and expected careers (Questions C6 and C7 in the follow-up survey). 12 Most of these 
measures have been validated by previous research (Kirschner 1989; Glanville and Wildhagen 
2007; Ogbu 2003; Nichols 2003; Ehrenberg et al. 1991; Balfanz and Herzog 2006; Smerdon 
1999; Coleman and Delaire 2003), and some have been shown to be correlated with later 
measures of success (Buchanan 1998). We also tested to see whether the estimated impacts of 
RTS on these outcomes varied with baseline student characteristics such as student test scores 
and parent education. 

Table IV. 1 presents psychometric properties for each of the confirmatory outcome measures 
and most of the exploratory outcome measures. The table does not present psychometric 
properties for the individual items we used to create constructs measuring school absences and 
negative behaviors, career exploration with others, or career exploration efficacy. The 
“reasonably ambitious and realistic career expectations” outcome is also not included in Table 
IV. 1; this measure is based on open-ended data from the follow-up student survey. See Appendix 
B for a description of the procedures we used to code these data. 



1 1 Career exploration behavior might be seen as a measure of program fidelity rather than as an outcome. 

12 Questions A6, A7, and Cll contain multiple sub-questions. We took the average response across the sub- 
questions for our main analysis. We then compared outcomes for treatment and control students on each individual 
sub-question. 



24 




TABLE IV. 1 



PSYCHOMETRIC PROPERTIES OF OUTCOME MEASURES 





Number of 










Cronbach’s 


Outcome 


Items 


Mean 


SD 


Minimum Value 


Maximum Value 


Alpha 


Motivation to go to school to learn job skills 
(“I attend school to learn skills for a job. ”) 


1 


3.26 


0.75 


1 (strongly disagree) 


4 (strongly agree) 


n/a 


Learning and study habits (for example, “I 


6 


4.01 


0.73 


1 (never) 


5 (always/almost 


0.85 


stick with a class assignment until it is done. ”) 




always) 


School absences and negative behaviors (for 
example, “How many times were you late for 
school?”) 


5 


1.78 


0.55 


1 (never) 


4 (10+ times) 


0.69 


Career exploration with parents (for example, 
“I discussed whether to go to college with a 
parent. ”) 


5 


3.27 


1.04 


1 (never) 


5 (very often) 


0.88 


Career exploration with school staff (for 
example, “I discussed whether to go to college 
with a teacher/school staff. ”) 


5 


2.77 


1.13 


1 (never) 


5 (very often) 


0.93 


School engagement (“I go to school because 
the subjects I’m taking are interesting. ”) 


1 


2.49 


0.79 


1 (strongly disagree) 


4 (strongly agree) 


n/a 


Importance of grades (“How important are 
good grades to you ? ”) 


1 


3.42 


0.75 


1 (not at all 
important) 


4 (very important) 


n/a 


Career exploration efficacy (for example, “I 
know what is required to succeed in different 
careers. ”) 


4 


3.28 


0.54 


1 (strongly disagree) 


4 (strongly agree) 


0.81 



Source: Mathematica student surveys administered in spring 2009. 
SD = standard deviation. 
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C. IMPACT ESTIMATION 



1. Description of Benchmark Model 

To estimate impacts, we use linear regressions of student outcomes at the end of grade 8 on 
an indicator of treatment status and control for student baseline test scores and other 
characteristics. In all analyses, standard errors are adjusted for the clustering of students within 
schools and for heteroskedasticity. The following equation illustrates the model: 



(1) Y| — a + Ti* pi + Xi’p 2 + £cs + ei, 
where 



Yi = the outcome variable for individual i 
Ti = treatment status (1 if treatment, 0 if control) 

Xj = student and school baseline characteristics (grade 6 test scores, race/ethnicity, gender, 
age, parent education, main language spoken at home, Internet access at home, school 
average 6th grade test scores, and school average free lunch) 
s cs = unobserved school- or cohort-level factors 
ej = unobserved student-level factors 
a, Pi, P 2 = parameters to be estimated 



This work is complicated by four important factors — student mobility, the possibility of 
contamination of the control group by the treatment group through the sharing of information 
across grade levels within schools, student nonresponse to the surveys, and missing baseline test 
score data. All of these factors could affect estimated impacts. We deal with these problems in 
the analyses, as explained below. 



2. Student Mobility 

Students often change schools within districts and sometimes move between districts. RTS 
could affect student mobility if, for example, the presence of the RTS intervention encourages 
some students to remain in their current school and perhaps even attracts others to that school. 
According to student rosters provided by the schools, 13 students who were originally enrolled in 
treatment schools in fall 2007 moved to control schools during the course of the study. Likewise, 
nine students who were originally enrolled in control schools in fall 2007 moved to treatment 
schools during the course of the study. 

Students who started grade 7 in a treatment school could have skipped a grade or been held 
back during the course of the study, meaning that they would enter or exit the grade receiving the 
RTS intervention. Likewise, students who started grade 7 in a control school could have skipped 
a grade or been held back during the course of the study, meaning that they would enter or exit 
the control group. According to the student rosters provided by schools, this type of movement 
did not occur during the course of the study. 
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We address student mobility in our analyses in a number of ways. First, students are 
analyzed based on their treatment/control group status at the beginning of the study in fall 2007. 
Thus we use an “Intent to Treat” analysis. 13 Second, we use the baseline test score data to test 
whether the RTS intervention appears to have any impact on student mobility. We found no 
statistically significant difference between the test scores of the students who moved into the 
treatment group from the control group and those who moved into the control group from the 
treatment group. Third, we tested to see if the total number of mobile students differed between 
the treatment and control groups. As Table III. 3 (above) shows, the fractions of students who 
moved out of the treatment and control schools were very similar (10 and 9 percent 
respectively). 14 



3. Contamination 

Our estimation method relies heavily on the assumption that the control group is not affected 
by the presence of the RTS program in its school. This assumption could be violated in at least 
two ways. 



1. Schoolwide Elements. The control group schools were allowed to have a younger 
cohort of grade 7 students participate in RTS starting in the 2008-2009 school year. 
Data from this younger cohort of students are not included in the current report. 
However, the control group students in the current study were in schools that offered 
RTS to younger students during the 2008-2009 school year and thus were exposed to 
any schoolwide elements of the RTS intervention during this period. Counselors (and 
other staff) at these schools were likely to learn at least some information from the 
RTS facilitator and may have used this information to improve outcomes for control 
group students. The presence of the RTS program may have also enabled the 
counselors to spend more time with the control group students if they felt that the 
treatment group students needed less of their time. Finally, the RTS organization 
purchased a license for the Career Cruising website that provided students with 
career planning information. Members of the control group as well as the treatment 
group could have accessed the website. 

2. Siblings with RTS Exposure. Students in the control group during the 2008-2009 
school year may have had siblings in enrolled in grade 7 in the same school who 
were receiving the RTS intervention. These siblings could share information 
obtained from RTS, either directly or through their parents, who received an RTS 
newsletter. According to student responses on the spring 2009 follow-up survey, 17 
percent (n = 109) of the 631 control students who responded to the survey question 
reported having a sibling or someone else in their home who was currently attending 
grade 7 at their school. 



13 Students who were not in the study in fall 2007 but who later moved into a study school are omitted from all 
analyses because their movement into these schools could have been influenced by the presence of RTS. 

14 The difference is not statistically significant at the 0.10 level. 
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RTS staff members believe that these spillover effects were negligible because the program 
was implemented in only one grade at a time requiring any sharing of information to occur 
across grade levels. Because some sharing of information is likely for the reasons given above, 
we are, in effect, estimating the impacts of this type of sharing for the control group compared to 
a far more intensive intervention for the treatment group that includes 45 minutes per week of 
instruction. If the information shared across grades creates impacts comparable to the intensive 
RTS intervention, this would suggest that a less time-intensive intervention might be sufficient to 
obtain similar impacts. Thus, bias caused by these types of spillovers will not invalidate the 
usefulness of the results of this study. That said, we did run models controlling for whether a 
child in the control group had a sibling in the treatment group. Those results are described below. 



4. Nonresponse to Student Surveys 

We deal with nonresponse bias in two ways. First, we include control variables in our 
analyses. These control variables adjust for any differences between the treatment and control 
groups in observable characteristics that might be due to differential nonresponse. Second, we 
weight the data so that the treatment and control groups are balanced with respect to the blocks 
of matched schools. Following is a more detailed description of how we created our weights. 

As explained above, schools were matched into blocks with two or three schools per block. 
One school was randomly selected from each block to be in the treatment group. The remaining 
schools were put in the control group. The numbers of responding students in these schools were 
used to create the weights as follows: 

(2) Treatment Group Weights = NBR / NBR t 

(3) Control Group Weights = NBR / NBR Ci 
where 

NBR = total student respondents in block (across all schools) 

NBR t = total student respondents in treatment schools in block 

NBR C = total student respondents in control school in block (each block has only one control 
school). 

Students who are non-respondents are omitted from these calculations. This includes all 
students in the non-responding schools as well as the non-responding students within the 
responding schools. Thus our estimates are not representative for those students. 

Table IV. 2 provides characteristics of the students used in our analyses, both weighted and 
unweighted. We consider two groups of students (those with full control variables and the larger 
set including some students with missing control variables). As Table IV. 2 illustrates, the 
characteristics of the students used in our analyses are generally similar regardless of which 
group is used or whether the results are weighted or unweighted. 

We calculated separate weights for each of the two different analysis samples. The first 
sample, referred to as the full analysis sample, is used in the analyses without control variables, 
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such as those reported in Table V.2 and Table V.4. The second sample, referred to as the control 
variables analysis sample, is used in the analyses that include control variables, such as those 
reported in Table V.3 and Table V.5. The weights for the control variables analysis sample are 
calculated using only cases that have at least one of the outcome variables and all of the control 
variables. The weights for the full analysis sample are calculated using a larger set of cases, still 
limited to those with at least one outcome but including cases with missing control variables. 

TABLE IV.2 

SAMPLE CHARACTERISTICS 





Full Analysis Sample 


Control Variables Analysis Sample 


Variable 


Unweighted 


Weighted 


Unweighted 


Weighted 


Female 


50.8% 


50.9% 


52.1% 


52.1% 


Race/Ethnicity 










White 


90.8% 


91.4% 


91.2% 


91.8% 


Black or African American 


1.3% 


1.1% 


1.3% 


1.0% 


Hispanic or Latino/Latina 


0.6% 


0.6% 


0.6% 


0.6% 


Asian 


0.4% 


0.4% 


0.3% 


0.3% 


American Indian or Alaska Native 


0.1% 


0.1% 


0.2% 


0.1% 


Other/mixed/unknown 


6.8% 


6.4% 


6.4% 


6.1% 


Mother’s Highest Level of Education 










At least high school diploma 


68.6% 


67.5% 


69.4% 


68.2% 


At least bachelor’s degree 


24.5% 


24.1% 


24.1% 


23.8% 


Father’s Highest Level of Education 










At least high school diploma 


62.0% 


61.0% 


62.6% 


61.5% 


At least bachelor's degree 


15.5% 


14.4% 


15.6% 


14.5% 


English Spoken at Home 


98.1% 


98.2% 


98.7% 


98.9% 


Over Age for Grade 


2.6% 


2.8% 


2.2% 


2.2% 


Computer Access at Home 


82.5% 


81.5% 


82.6% 


81.7% 


Baseline Math Z-score 


0.09 


0.08 


0.09 


0.09 


Baseline Reading Z-score 


0.12 


0.09 


0.13 


0.10 


Sample Size 


1,444 


1,305 



Source: Mathematica student surveys administered to all study students in fall 2007 and spring 2009; if data were 
missing for the baseline survey, we used the follow-up survey data from 2009 covering the same 
information. Baseline test score data are from grade 6 standardized math and English/language arts tests 
from spring 2007. 



5. Missing Baseline Test Score Data 

Some schools provided proficiency level data for students but not their scaled test scores. 
We include these students in the regression analysis by using their proficiency level to impute 
their scaled score, separately for math and reading. This imputation procedure involves 
regressing scaled scores on proficiency-level indicators, as well as other student-level control 
variables, for students with non-missing scaled scores. This regression is then used to predict 
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scaled score values for students where those scores are missing, based on their proficiency level 
and other observable characteristics. We imputed test scores in this way for 155 out of the 1,305 
students in the control variables analysis sample. 

Some students are omitted from the control variables analysis sample because of missing 
data. From the full analysis sample of 1,444 students with non-missing values for at least one of 
the outcome variables, 58 are omitted because they are missing the race/ethnicity variable, 78 are 
omitted because they lack the data necessary for calculating their math or reading z-score 
(because they are missing both scale scores and proficiency levels), and 3 additional students are 
omitted because they are missing one of the other control variables. This leaves a control 
variables analysis sample of 1,305 students. 15 



15 The control variables analysis sample sizes vary slightly by outcome measure, from 1,287 to 1,299 students. 
All students included have data on at least one outcome, but for each outcome measure there are some students in 
the analysis sample with missing data. Data on parent education was missing for a quarter of the sample. These 
students are included in all of the tables in this report. The missing values were included in the omitted category for 
parent education. We also ran the results for the last two columns of Table V.3 without these observations. The 
results were substantively the same. The only estimated effect of Roads to Success that was statistically significant 
was for the same outcome as in Table V.3 (school absences and negative behaviors). That estimate remained 
negative and statistically significant at the 10% level but not at the 5% level. None of the estimated impacts of 
Roads to Success in Table V.3 changed in a statistically significant way. These results are available upon request. 
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V. IMPACT FINDINGS 



We estimate intervention impacts by comparing outcomes for the treatment and control 
groups using multivariate regression methods. The impact analysis focuses on two types of 
outcomes — a confirmatory set, used to test how well our pre-specified hypotheses are supported 
by the data and an exploratory set, used to test additional hypotheses that might be the basis for 
more rigorous testing in later studies. We first present simple differences in means, followed by 
the regression analysis controlling for baseline test scores and other observable characteristics 
and a weighted analysis to control for potential bias due to nonresponse. 

Table V.l shows unweighted descriptive characteristics of the control variables analysis 
sample, separately for treatment and control students. Although there are differences, none of 
them are statistically significant. 



TABLE V.l 

TESTS OF EQUIVALENCE FOR ANALYSIS SAMPLE 





Treatment 


Control 




Variable 


Students 


Students 


Difference 


Female 


51.4% 


53.0% 


-1.6 


Race/Ethnicity 








White 


91.9% 


90.3 % 


1.6 


Black or African American 


1.2% 


1.4% 


-0.2 


Hispanic or Latino/Latina 


0.4% 


0.9% 


-0.5 


Asian 


0.3% 


0.3% 


0.0 


American Indian or Alaska Native 


0.3% 


0% 


0.3 


Other/mixed/unknown 


5.9% 


7.1% 


-1.2 


Mother’s Highest Level of Education 








At least high school diploma 


70.6% 


67.9% 


2.7 


At least bachelor’s degree 


24.7 % 


23.4% 


1.3 


Father’s Highest Level of Education 








At least high school diploma 


63.9% 


61.0% 


2.9 


At least bachelor’s degree 


16.3 % 


14.6% 


2.7 


English Spoken at Home 


98.5 % 


99.0% 


-0.5 


Over Age for Grade 


1.6% 


2.9% 


-1.3 


Computer Access at Home 


82.8% 


82.4 % 


-1.6 


Baseline Math Z-score 


0.06 


0.13 


-0.07 


Baseline Reading Z-score 


0.13 


0.12 


0.01 


Sample Size 


728 


577 





Source: Mathematica student surveys administered to all study students in fall 2007 and spring 2009; if data were 
missing for the baseline survey, we used the follow-up survey data from 2009 covering the same 
information. Baseline test score data are from grade 6 standardized math and English/language arts tests 
from spring 2007. 

Notes: These results are based on unweighted data. None of the differences are statistically significant at the 0.05 

level using a two-tailed test. 
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A. CONFIRMATORY OUTCOMES 



Table V.2 shows the differences between treatment and control group students on the three 
confirmatory outcomes: (1) motivation to go to school to leam job skills; (2) learning, study 
habits, and preparation; and (3) school absences and negative behaviors. Observations are 
weighted to adjust for differences in nonresponse rates, and standard errors are adjusted for the 
clustering of students within schools. 16 This comparison of group means for the full analysis 
sample reveals no statistically significant differences in average responses between treatment and 
control group students. 



TABLE V.2 

SAMPLE DIFFERENCES ON GRADE 8 CONFIRMATORY OUTCOMES 



Outcome 


Sample 

Size 


Treatment 

Mean 


Control 

Mean 


Difference 


/?- value 


Motivation to go to school to learn job 
skills 


1,420 


3.201 


3.319 


-0.118 


0.101 


Learning and study habits and preparation 


1,435 


3.983 


4.038 


-0.055 


0.394 


School absences and negative behaviors 


1,434 


1.779 


1.817 


-0.038 


0.431 



Source: Mathematica student surveys administered in spring 2009. 

Notes: Observations are weighted to adjust for nonresponse. Standard errors are adjusted for clustering of students 

by school. The standard deviations of the outcomes are 0.75, 0.73, and 0.55 respectively. 

*Significantly different from 0 at the .10 level, two-tailed test. 

**Significantly different from 0 at the .05 level, two-tailed test. 

***Significantly different from 0 at the .01 level, two-tailed test. 



Table V.3 presents results from a linear regression analysis comparing outcomes for 
treatment and control students for the control variables analysis sample, including baseline test 
scores as a control variable. The first column of results does not include the control variables in 
the model but does adjust for clustering of students within schools. There is a statistically 
significant difference at the 0.10 level between treatment and control groups on the first outcome 
measure, with the control group students reporting greater motivation to go to school to leam job 
skills. Differences on the other two confirmatory outcome measures are not statistically 
significant. 

The second column of results in Table V.3 is from a regression model that includes controls 
for baseline math and reading test scores, as well as for other observable student and school 
characteristics. When these controls are included, none of the differences are statistically 
significant. 

Because the analysis sample does not include every student from the original baseline 
sample, due to nonresponse and missing data issues, it is possible that the results are biased due 



16 See Chapter III for a discussion of the method for constructing the nonresponse weights. Because the 
analyses performed in tables V.2 and V.4 do not require control variables, whereas the analyses in tables V.3 and 
V.5 do, different weights were constructed for each pair of tables. 
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to differential response between treatment and control groups. To account for this possibility, we 
ran an analysis weighting each observation to adjust for nonresponse. 17 The results of this 
analysis are shown in the last column of Table V.3. Once we control for nonresponse, none of 
the confirmatory outcome measures are statistically significant at the .05 level. Additionally, we 
tested the joint significance of all three confirmatory outcomes, and they are not jointly 
significant at the 0.10 level. There is statistically significant difference between treatment and 
control students on school absences and negative behaviors at the 0.10 level, with treatment 
group students reporting fewer absences and negative behaviors. 

TABLE V.3 

ESTIMATED IMPACTS ON GRADE 8 CONFIRMATORY OUTCOMES (p- VALUES) 





Sample 

Size 


Unweighted, No 
Control Variables 


Unweighted, 
with Control 
Variables 


Weighted, with Control 
Variables 


Motivation to go to school to learn 
job skills 


1,287 


-0.117* (0.092) 


-0.110 

(0.122) 


-0.083 (0.267) 


Learning and study habits and 
preparation 


1,299 


-0.029 (0.631) 


-0.070 

(0.289) 


-0.046 (0.524) 


School absences and negative 
behaviors 


1,298 


-0.052 (0.301) 


-0.052 

(0.194) 


-0.064* (0.087) 



Source: Mathematica student surveys administered in fall 2007 and spring 2009. 

Notes: The control variables include grade 6 test scores by subject (math and reading); race/ethnicity indicators 

(Black/African American, Hispanic, Asian, American Indian, other race); gender; age; parent education 
separately for the mother and father (has at least a high school diploma, has at least a bachelor’s degree); 
primary language at home is not English; internet access at home; school average grade 6 test scores by 
subject; and school average free lunch. The standard deviations of the outcomes are 0.75, 0.73, and 0.55 
respectively. 

* Significantly different from 0 at the .10 level, two-tailed test. 

**Significantly different from 0 at the .05 level, two-tailed test. 

***Significantly different from 0 at the .01 level, two-tailed test. 



Although the confirmatory outcome measures are not statistically significant by themselves, 
it is possible that there are significant differences in the outcomes by different student and school 
characteristics. To test for this, we interacted the treatment indicator with each of the control 
variables and then ran the weighted model including these interactions as additional control 
variables. The combined interactions are not jointly significant at the 0.10 level for any of the 
outcome measures, indicating that there are not statistically significant differences in the 
treatment effect by student or school characteristics. 



17 This is the specification from the design report (Chaplin et al. 2009). 

18 The coefficient estimates for all variables in this model, including the control variables, are presented in 
Appendix D. 
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As discussed above, there is some chance of spillover between the treatment and control 
groups if a control group child has a sibling in the treatment group. Consequently we estimated 
impacts of RTS on the three main confirmatory outcomes described above using the same 
methods as were used for the results in Table V.3, but adding in a covariate to identify if a 
control group child had a sibling in the treatment group. The results (coefficient estimates and 
standard errors) were similar to those shown in Table V.3 except that the standard error for the 
last outcome dropped and that result became statistically significant at the 10 percent level. The 
coefficient estimate on the RTS treatment variable was -0.068 in this regression as compared to 
-0.064 in Table V.3. 



B. EXPLORATORY OUTCOMES 

In our exploratory analyses we start by breaking out the individual items used to create the 
school absences and negative behaviors measure analyzed as a confirmatory outcome. We then 
look at five additional variables. 



1. School Absences and Negative Behaviors 

The construct measuring school absences and negative behaviors consisted of five individual 
items. Because the composite score may mask impacts for individual items, we compared 
outcomes for treatment and control students on each individual item (Table V.4). There is a 
statistically significant difference between treatment and control students on detentions, with the 
control group students reporting more detentions (at the 0.01 level). 

TABLE V.4 

SAMPLE DIFFERENCES ON SCHOOL ABSENCES AND NEGATIVE BEHAVIORS 



Outcome 


Sample 

Size 


Treatment 

Mean 


Control 

Mean 


Difference 


p- value 


How many times. . . 












Were you late for school? 


1,431 


1.944 


1.881 


0.063 


0.473 


Did you cut or skip classes? 


1,428 


1.322 


1.230 


0.092 


0.102 


Were you absent from school? 


1,428 


2.620 


2.685 


-0.065 


0.414 


Were you sent out of class for bad 
behavior? 


1,427 


1.425 


1.441 


-0.016 


0.824 


Were you given a detention? 


1,428 


1.511 


1.797 


-0.286 *** 


0.004 



Source: Mathematica student surveys administered in spring 2009. 

Notes: Scoring scale: (1) never, (2) l^t times, (3) 5-9 times, and (4) 10 or more times. These results are based on 

unweighted data. Standard errors are adjusted for clustering of students by school. The standard deviations 
of the outcomes are 0.87, 0.65, 0.89, 0.79, and 0.88 respectively. 

* Significantly different from 0 at the .10 level, two-tailed test. 

**Significantly different from 0 at the .05 level, two-tailed test. 

***Significantly different from 0 at the .01 level, two-tailed test. 
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Table V.5 presents results for the control variables analysis sample. The first column of 
results does not include the control variables in the model but does adjust for clustering of 
students within schools. The estimated impact on the detention variable is still statistically 
significant (at the 0.01 level), as it was in Table V.4. Estimated impacts on the other four 
measures are not statistically significant. The second column of results in Table V.5 includes 
controls for baseline math and reading test scores as well as for other observable student 
characteristics. Once these control variables are included, the difference on cutting/skipping 
classes becomes significant (at the 0.10 level). The third column of results adds weights to adjust 
for nonresponse. After weighting, cutting/skipping classes, absences, and detentions are 
significant, at the 0.10, .05, and 0.10 levels, respectively. Thus, RTS seems to be helpful in two 
areas (by reducing absenteeism and detentions) but potentially harmful in another (by increasing 
the rate at which students skip class). 



TABLE V.5 

ESTIMATED IMPACTS ON SCHOOL ABSENCES 
AND NEGATIVE BEHAVIORS (p- VALUES) 





Sample 

Size 


Unweighted, No Control 
Variables 


Unweighted 

with 

Control 

Variables 


Weighted with Control 
Variables 


How many times. . . 

Were you late for school? 


1,295 


0.065 (0.466) 


0.035 

(0.736) 


-0.020 (0.831) 


Did you cut or skip classes? 


1,293 


0.079(0.133) 


0.115* 

(0.055) 


0.124* (0.071) 


Were you absent from school? 


1,293 


-0.073 (0.388) 


-0.105 

(0.110) 


-0.124** (0.030) 


Were you sent out of class for bad 
behavior 


1,291 


-0.028 (0.670) 


-0.063 

(0.270) 


-0.088 (0.124) 


Were you given a detention? 


1,293 


-0.292*** (0.002) 


-0.226** 

(0.021) 


-0.200* (0.058) 



Source: Mathematica student surveys administered in fall 2007 and spring 2009. 

Notes: The control variables include grade 6 test scores by subject (math and reading); race/ethnicity indicators 

(Black/African American, Hispanic, Asian, American Indian, other race); gender; age; parent education 
separately for the mother and father (has at least a high school diploma, has at least a bachelor’s degree); 
primary language at home is not English; internet access at home; school average grade 6 test scores by 
subject; and school average free lunch. The standard deviations of the outcomes are 0.87, 0.65, 0.89, 0.79, 
and 0.88 respectively. 

* Significantly different from 0 at the .10 level, two-tailed test. 

** Significantly different from 0 at the .05 level, two-tailed test. 

***Significantly different from 0 at the .01 level, two-tailed test. 



2. Career Exploration and School Engagement 

Table V.6 shows the differences between treatment and control group students on the five 
additional exploratory outcome measures: career exploration with parents, career exploration 
with school staff, school engagement, importance of grades, and career exploration efficacy. This 
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comparison of weighted group means reveals a statistically significant difference between 
treatment and control groups, with treatment group students reporting higher average values on 
career exploration with school staff (at the 0.01 level). For the other outcomes, there are no 
significant differences in average responses between treatment and control group students. 

TABLE V.6 

SAMPLE DIFFERENCES ON GRADE 8 EXPLORATORY OUTCOMES 



Outcome 


Sample 

Size 


Treatment Mean 


Control Mean 


Difference 


p- value 


Career exploration with parents 


1,435 


3.262 


3.242 


0.020 


0.862 


Career exploration with school staff 


1,435 


3.011 


2.494 


0.517 *** 


0.004 


School engagement 


1,426 


2.455 


2.549 


-0.094 


0.237 


Importance of grades 


1,431 


3.400 


3.415 


-0.015 


0.846 


Career exploration efficacy 


1,429 


3.299 


3.243 


0.056 


0.151 



Source: Mathematica student survey administered to study students in spring 2009. 

Notes: Observations are weighted to adjust for nonresponse. Standard errors are adjusted for clustering of students 

by school. The standard deviations of the outcomes are 1.04, 1.13, 0.79, 0.75, and 0.54 respectively. 

* Significantly different from 0 at the .10 level, two-tailed test. 

**Significantly different from 0 at the .05 level, two-tailed test. 

***Significantly different from 0 at the .01 level, two-tailed test. 



Table V.7 presents results for the control variables analysis sample. The first column of 
results does not include the control variables in the model, but does adjust for clustering of 
students within schools. Once this adjustment is made, the difference in career exploration with 
school staff (at the 0.05 level) is still statistically significant and career exploration efficacy (at 
the 0.10 level) is still significant. Differences on the other three exploratory outcome measures 
are not statistically significant. 

The second column of results in Table V.7 includes controls for baseline math and reading 
test scores as well as for other observable student characteristics. The results change very little 
when these controls are included, with the only difference being that career exploration efficacy 
is no longer significant. The third column of results adds weights to adjust for nonresponse. After 
weighting, both career exploration with school staff and career exploration efficacy are 
significant, at the 0.01 and 0.10 levels respectively. Together, estimated impacts of RTS on the 
five exploratory outcomes in Table V.7 are jointly significant at the 0.05 level. 
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TABLE V.7 



ESTIMATED IMPACTS ON GRADE 8 EXPLORATORY OUTCOMES (p- VALUES) 





Sample 

Size 


Unweighted, No 
Control Variables 


Unweighted with 
Control Variables 


Weighted with Control 
Variables 


Career exploration with 
parents 


1,291 


-0.016 (0.880) 


-0.024 (0.841) 


0.007 (0.956) 


Career exploration with 
school staff 


1,297 


0.456** (0.012) 


0.558*** (0.010) 


0.596*** (0.006) 


School engagement 


1,299 


-0.076 (0.265) 


-0.065 (0.300) 


-0.054 (0.400) 


Importance of grades 


1,299 


-0.032 (0.599) 


-0.048 (0.487) 


-0.032 (0.674) 


Career exploration efficacy 


1,294 


0.062* (0.090) 


0.049(0.163) 


0.055* (0.068) 



Source: Mathematica student surveys administered in fall 2007 and spring 2009. 

Notes: The control variables include grade 6 test scores by subject (math and reading); race/ethnicity indicators 

(Black/African American, Hispanic, Asian, American Indian, other race); gender; age; parent education 
separately for the mother and father (has at least a high school diploma, has at least a bachelor’s degree); 
primary language at home is not English; internet access at home; school average grade 6 test scores by 
subject; and school average free lunch. The standard deviations of the outcomes are 1.04, 1.13, 0.79, 0.75, 
and 0.54 respectively. 

* Significantly different from 0 at the .10 level, two-tailed test. 

**Significantly different from 0 at the .05 level, two-tailed test. 

***Significantly different from 0 at the .01 level, two-tailed test. 



3. Detailed Examination of Career Exploration with Others 

We also looked more directly at how much the students were talking to their parents and 
school staff about future careers. As Table V.8 shows, the students in the control group, without 
RTS, were talking to their parents more than their teachers about careers. RTS increased how 
much students talk to school staff, but even the students in the treatment group spoke more to 
their parents than to school staff about “whether to go to college,” “which college,” to attend and 
“careers.” We also created variables that looked at the maximum levels of how much the 
students spoke to either their parents or school staff. 19 RTS had no statistically significant 
impacts on these variables. 



' These variables are equal to the maximum values of the corresponding variables for parents and school staff. 



A6 and A7. 
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TABLE V.8 



SAMPLE DIFFERENCES ON CAREER EXPLORATION WITH OTHERS 





Treatment Means 


Control Means 


Outcome 


Sample 

Size 


With 

Parent 


With 

Teacher 


Difference 

(Teacher- 

Parent) 


With 

Parent 


With 

Teacher 


Difference 

(Teacher- 

Parent) 


In the last year, how 
often have you 
discussed the 
following . . . 
















College entrance exams 


1,428 


2.51 


2.59 


0.08 


2.49 


2.27 


-0.22** 


Whether to go to college 


1,426 


3.68 


3.10 


-0.58*** 


3.71 


2.53 


-1 18*** 


What college to choose 


1,426 


3.22 


2.96 


-0.26*** 


3.18 


2.20 


-0 98*** 


College majors and 
programs 


1,413 


3.08 


3.03 


-0.05 


3.09 


2.59 


-0.50*** 


Possible jobs or careers 


1,425 


3.84 


3.32 


-0.52*** 


3.90 


2.95 


-0.95*** 



Source: Mathematica student surveys administered in spring 2009. 

Notes: Survey asked how often students discussed topic with teacher or other school staff. Scoring scale: (1) 

never, (2) not very often, (3) sometimes, (4) often, and (5) very often. These results are based on 
unweighted data. 

*Significantly different from 0 at the .10 level, two-tailed test. 

**Significantly different from 0 at the .05 level, two-tailed test. 

***Significantly different from 0 at the .01 level, two-tailed test. 



4. Detailed Examination of Career Exploration Efficacy 

The construct measuring career exploration efficacy consisted of four individual items. 
Because we were interested in the effect of RTS on each of these indicators, we compared 
outcomes for treatment and control students on each individual item (see Table V.9). Treatment 
students were more likely to report knowing how to find out about what types of jobs are best for 
them (statistically significant at the 0.01 level) and to report knowing what is required to succeed 
in different careers (significant at the 0.10 level). 
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TABLE V.9 



SAMPLE DIFFERENCES ON CAREER EXPLORATION EFFICACY 



Outcome 


Sample 

Size 


Treatment 

Mean 


Control 

Mean 


Difference 


p-value 


I know what is required to succeed in different 
careers. 


1,422 


3.216 


3.125 


0.091 * 


0.053 


I know how to find out about what types of jobs 
are best for me. 


1,424 


3.270 


3.133 


0.137 *** 


0.001 


I have a good idea about the kinds of jobs I would 
be good at. 


1,423 


3.443 


3.443 


0.000 


0.999 


I will be able to overcome barriers that stand in 
the way of achieving my career goals. 


1,424 


3.311 


3.314 


-0.003 


0.942 



Source: Mathematica student surveys administered in spring 2009. 

Notes: Scoring scale: (1) strongly disagree, (2) disagree, (3) agree, and (4) strongly agree. These results are based 

on unweighted data. Standard errors are adjusted for clustering of students by school. The standard 
deviations of the outcomes are 0.66, 0.70, 0.64, and 0.66 respectively. 

* Significantly different from 0 at the .10 level, two-tailed test. 

**Significantly different from 0 at the .05 level, two-tailed test. 

***Significantly different from 0 at the .01 level, two-tailed test. 



Table V.10 presents results on career exploration efficacy for the control variables analysis 
sample. The first column of results does not include the control variables in the model but does 
adjust for clustering of students within schools. Once this adjustment is made, the differences in 
knowing how to find out about what types of jobs are best (at the 0.10 level) and in knowing 
what is required to succeed in different careers (statistically significant at the 0.01 level) are 
statistically significant. Differences on the other two outcome measures are not statistically 
significant. The second column of results in Table V.10 includes controls for baseline math and 
reading test scores as well as for other observable student characteristics. The results change very 
little when these controls are included, with the only difference being that knowing what is 
required to succeed in different careers is no longer statistically significant. The third column of 
results adds weights to adjust for nonresponse. After weighting, both knowing what is required to 
succeed in different careers and knowing how to find out about what types of jobs are best are 
statistically significant at the 0.10 and 0.01 levels, respectively. 

We were particularly interested in the possibility that RTS might give students the sense that 
they can overcome barriers associated with career success (the fourth question in this set of 
survey items). For this reason, we also estimated impacts of RTS on whether students reported 
agreeing or strongly agreeing to this question. The estimated impact of RTS remained not 
statistically significant with a coefficient estimate of -0.013 and a /;- value of 0.71. Perhaps of 



~ A binary variable ( 1 = agree or strongly agree, 2 = disagree or strongly disagree) was used as the outcome in 
this analysis. We used the same linear regression model that was used for results presented in the last column of 
Table V.9. 



39 




greater importance, more than 90 percent of both the treatment and control groups reported 
agreeing or strongly agreeing with this statement. 

TABLE V.10 

ESTIMATED IMPACTS ON CAREER EXPLORATION EFFICIENCY (p- VALUES) 





Sample 

Size 


Unweighted, No 
Control Variables 


Unweighted, 
with Control 
Variables 


Weighted, with 
Control Variables 


I know what is required to succeed in 
different careers. 


1,288 


0.096* (0.057) 


0.077 (0.120) 


0.079* (0.057) 


I know how to find out about what 
types of jobs are best for me. 


1,290 


0.147*** (0.000) 


0.147*** (0.001) 


0.161*** (0.000) 


I have a good idea about the kinds of 
jobs I would be good at. 


1,289 


0.008 (0.836) 


-0.007 (0.847) 


-0.017(0.641) 


I will be able to overcome barriers that 
stand in the way of achieving my 
career goals. 


1,291 


0.009 (0.842) 


-0.008 (0.853) 


0.012 (0.764) 



Source: Mathematica student surveys administered in fall 2007 and spring 2009. 

Notes: The control variables include grade 6 test scores by subject (math and reading); race/ethnicity indicators 

(Black/African American, Hispanic, Asian, American Indian, other race); gender; age; parent education 
separately for the mother and father (has at least a high school diploma, has at least a bachelor’s degree); 
primary language at home is not English; internet access at home; school average grade 6 test scores by 
subject; and school average free lunch. The standard deviations of the outcomes are 0.66, 0.70, 0.64, and 
0.66 respectively. 

* Significantly different from 0 at the .10 level, two-tailed test. 

** Significantly different from 0 at the .05 level, two-tailed test. 

***Significantly different from 0 at the .01 level, two-tailed test. 



5. Desired and Expected Careers 

We asked students to report on four careers in the survey, three that they wanted to have and 
one that they expected to have. As discussed above, we coded these careers based on how 
realistic and reasonably ambitious they were. The findings were remarkably similar across all 
four careers. Among treatment and control students, only about 10 percent reported careers that 
were unrealistic (such as professional sports, actor, model). 21 Fewer than one percent reported 
plans to be a homemaker and less than 1 percent reported careers in the low ambition category 
(babysitter, lawn mowing, and so on). We found no evidence of statistically significant impacts 
of RTS on these outcomes. Although these results are encouraging in the sense that they suggest 
that these grade 8 students had reasonable career expectations, it should be noted that the fraction 
with low expectations might increase in higher grades as students become more aware of the 
possible barriers to success. 



21 Only four students reported plans to be “president” with no clarification. One other student planned to be 
president of a famous computer game company. 
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C. SUBGROUP ANALYSIS 



As part of our exploratory analyses we looked at impacts on our confirmatory outcomes 
separately by gender, test score (in quartiles separately for math and reading), parent education 
(less than high school, high school only, and greater than high school for each parent)"”, and for 
the half of students for whom the impacts of RTS on talking to school staff about careers was 
largest. 23 These breakouts suggested no clear findings, though the estimated impacts on talking 
to school staff about careers did appear to be somewhat higher for those with parents with lower 
levels of education. Results of these analyses are reported in Appendix E. 



D. ADDITIONAL ANALYSES 

We also estimated impacts on most of the remaining variables in our survey." These 
included the topics discussed with a parent or guardian (questions A4 and A6 on the follow-up 
survey); learning and study habits preparation (Bl); importance of various topics (Cl); expected 
education (C2); desired education (C3); reasons for possibly not achieving educational goals 
(C5); interest in careers (C61a, C62a, C63a, and C7a); likelihood of entering careers (C61b, 
C62b, C63b, and C7b); expectations for performance in careers (C61c, C62c, C63c, and C7c); 
education or training needed (C8); likelihood of obtaining needed education (C9); and reasons 
for not achieving goals (CIO). We found no consistent statistically significant results at the five 
percent level. 25 



r The subgroups were less than high school, high school only, and greater than high school. 

23 We identified this subgroup by estimating impacts of RTS on talking to a school staff member about careers 
and including interactions of RTS with the student test score and parent education variables described earlier in this 
paragraph. We then calculated the estimated impact of RTS on this outcome for each student and then separated 
students into the half with the largest impacts of RTS and the half with the smallest impacts of RTS . 

24 We excluded C4, which depended on the answer to another question, and section D, which covered student 
background information. 

25 Some coefficient estimates were statistically significant at the 10 percent level. Students were asked about 
how interested they were in four careers of their choosing. The estimated impact of Roads to Success on interest in 
the first career (C61a) was statistically significant and positive. However, a joint test across all four outcomes (C61a, 
C62a, C63a, and C7a) did not produce a statistically significant result. 
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VI. CONCLUSIONS 



Many experts suggest that middle and high school students need help in planning for their 
education and careers and that this work should start early, even before high school. The federal 
government has made a substantial investment in this area through programs such as GEAR UP. 
In this report we estimate impacts of RTS, a program that provides such training to students in 
grades 7 and 8, as well as in high school. We estimate impacts on three confirmatory 8th grade 
outcomes that we identified in the design report (Chaplin et al. 2009): 



• Motivation to go to school to learn job skills 

• Learning, study habits, and preparation 

• School absences and negative behaviors 



We find no statistically significant impacts on motivation to go to school or learning and 
study habits, though we do find a statistically significant impact at the .10 level for school 
absences and negative behaviors. These three confirmatory outcomes are not jointly significant 
at the 0.10 level and the point estimates all suggest effect sizes less than 0.20, which is small 
when compared to the standards described by Cohen (1988). 

We also did exploratory analyses on a number of other outcomes. Statistically significant 
impacts of RTS were found for a few of the individual items used to measure school absences 
and negative behaviors in weighted models controlling for test scores and observable student 
characteristics; students who received RTS reported fewer absences and detentions, but were 
more likely to report skipping class. Statistically significant impacts of RTS were also found for 
three career outcomes — career exploration with school staff (effect size approximately 0.50), 
knowing how to find out about what types of jobs are best for them (0.25), and knowing what is 
required to succeed in different careers (0.15). This suggests that RTS did increase the 
interactions these middle-school students had with school staff and their own career exploration 
efficacy. Overall, these results suggest that RTS does have not have clearly positive short-term 
impacts on the confirmatory outcomes we identified— motivation to go to school, learning and 
study habits, and school absences and negative behaviors. However, the program was designed 
to last for six years, not just two. Because the exploratory analysis shows that it did increase 
communication with school staff regarding career exploration, the effects on career related 
outcomes could differ in the long term compared to the short term. This finding suggests that it 
would be valuable to conduct a study of the full six-year version of the RTS program or of a 
similar program. 

The major limitation of this study is that it ended after only two years whereas the original 
study design called for following students for the full six years of their time in RTS. The study 
also had fairly low response rates. Only 67 percent of students provided follow-up data and only 
62 percent provided both baseline and follow-up data. The results do not appear sensitive to how 
we deal with these missing data, but bias is possible. In the future it might be preferable to try to 
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obtain outcome data from school districts or college entrance information from the National 
Student Clearinghouse to help reduce the nonresponse problems. 

A second limitation of this study is that the RTS intervention has been modified since this 
evaluation was done. For this reason, while the study provides good evidence of the impacts of 
the program at that time, it does limit the generalizability of the results to the current version of 
RTS. For example, in fall 2008, RTS added a new unit, “You Can Grow Your Intelligence,” to 
its grade 7 program. The unit challenges the idea of fixed intelligence, and is based on the 
“stereotype threat” research of Steele and Aronson (1995) and the malleability of intelligence 
research of Blackwell et al., (2007). The treatment students in the current study were in grade 8 
during the 2008-2009 school year and did not receive this new unit as part of their RTS 
experience. 

A third limitation of this study is that the results may not generalize to students most in need 
of these services. Although more than half of the students in these schools were eligible for free 
or reduced price lunches, the schools had few minority students. Of perhaps greater relevance, on 
average, students reported talking to their parents more than school staff about most issues 
regarding school and career planning and more than 90 percent reported agreeing or strongly 
agreeing with the statement that they “will be able to overcome barriers that stand in the way of 
achieving my career goals.” Also fewer than 10 percent of the treatment and control group 
students reported career aspirations or expectations that seemed too high and fewer than one 
percent reported careers that seemed too low. Thus, to the extent that the program was designed 
to improve communication with adults about career planning, improve career planning efficacy, 
or align student aspirations and expectations, these students may not have been in great need of 
these services. 

Although additional research on RTS may not be feasible because of funding cut-backs, the 
program staff created a great deal of documentation about the intervention, which similar 
interventions (such as those funded by GEAR UP) may be able to draw on. More generally, due 
to the paucity of strong research on interventions in this area, more studies using rigorous 
methods to estimate impacts of clearly defined interventions (such as RTS) are needed in order 
to determine what works and to inform future policy decisions. 
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APPENDIX A 

FOLLOW-UP STUDENT SURVEY INSTRUMENT 




f \ 

Student ID 

V ) 



MATHEMATICA 

Policy Research, Inc. 




Follow-up Survey 
Spring 2009 

INSTRUCTIONS 



• This survey asks about your involvement in school and the community, learning and 
study habits, and plans for the future. The survey should take about 30 minutes. 

• Mark only one answer for each question, unless the directions tell you to mark more 
than one answer. You may use a pen or pencil. 

• Your answers are very important to us. This is not a test. There are no right or 
wrong answers. You may skip any question you do not wish to answer. 

• If you have a question about the survey, raise your hand and someone will help you. 



6388-501 



A. SCHOOL AND COMMUNITY INVOLVEMENT 



A1 . How much do you agree or disagree with each of the following statements? 





MARK ONE FOR EACH ROW 




Strongly 

Disagree 


Disagree 


Agree 


Strongly 

Agree 


a. 1 go to school because 1 think the subjects I’m taking are 

interesting 


i □ 


2 □ 


3 □ 


4 □ 


b. 1 go to school because I’m learning skills that 1 will need for a job ... 


, □ 


2 □ 


3 □ 


4 □ 


c. 1 go to school because my parents or guardians expect me to 
succeed 


i □ 


2 □ 


3 □ 


4 □ 



A2. How important are good grades to you? 

MARK ONE ONLY 

1 □ Not important at all 

2 □ Somewhat important 

3 □ Important 

4 □ Very important 

A3. NOT IN THIS VERSION 



A4. In the last year, how often have you discussed the following with a parent or guardian? 





MARK ONE FOR EACH ROW 


Never 


Not Very 
Often 


Sometimes 


Often 


Very Often 


a. Selecting courses or programs at school 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 


b. School activities or events 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 


c. Topics you’ve studied in class 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 


d. Your grades 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 



A5. NOT IN THIS VERSION 
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A6. In the last year, how often have you discussed the following with a parent or guardian ? 





MARK ONE FOR EACH ROW 


Never 


Not Very 
Often 


Sometimes 


Often 


Very Often 


a. Taking college entrance exams (like the SAT or 
ACT) 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 


b. Whether to go to college 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 


c. What college to choose 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 


d. Different college majors and programs 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 


e. Possible jobs or careers 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 



A7. In the last year, how often have you discussed the following with one or more teachers or school staff 
(such as a guidance counselor)? 





MARK ONE FOR EACH ROW 




Never 


Not Very 
Often 


Sometimes 


Often 


Very Often 


a. Taking college entrance exams (like the SAT or 
ACT) 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 


b. Whether to go to college 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 


c. What college to choose 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 


d. Different college majors and programs 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 


e. Possible jobs or careers 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 
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B. LEARNING AND STUDY HABITS 



B1. Do you . . . 





MARK ONE FOR EACH ROW 


Never 


Not Very 
Often 


Sometimes 


Often 


Always or 
Almost 
Always 


a. stick with a class assignment or task until it is done? .... 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 


b. put in your best effort on class assignments, projects, 
and homework? 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 


c. ask a teacher or another student for help when you 
don’t understand an assignment? 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 


d. take part in class discussions or activities? 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 


e. come to your classes prepared with what you need 
(books, paper, and something to write with)? 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 


f. complete class assignments, projects, and homework 
on time? 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 





B2. NOT IN THIS VERSION 



B3. During the current school year . . . 





MARK ONE FOR EACH ROW 


Never 


1 -4 
Times 


5-9 

Times 


10 or More 
Times 


a. How many times were you late for school? 


i □ 


2 □ 


3 □ 


4 □ 


b. How many times did you cut or skip classes? 


i □ 


2 □ 


3 □ 


4 □ 


c. How many times were you absent from school? 


i □ 


2 □ 


3 □ 


4 □ 


d. How many times were you sent out of class for bad behavior? 


i □ 


2 □ 


3 □ 


4 □ 


e. How many times were you given a detention? 


i □ 


2 □ 


3 □ 


4 □ 
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C. PLANS FOR THE FUTURE 



Cl. How important is each of the following to you in your life? 





MARK ONE FOR EACH ROW 


Not 

Important 


Somewhat 

Important 


Important 


Very 

Important 


a. Being successful in my line of work 


i □ 


2 □ 


3 □ 


4 □ 


b. Having a happy family life 


i □ 


2 □ 


3 □ 


4 □ 


c. Having lots of money 


i □ 


2 □ 


3 □ 


4 □ 


d. Having strong friendships 


i □ 


2 □ 


3 □ 


4 □ 


e. Being able to find steady work 


i □ 


2 □ 


3 □ 


4 □ 


f. Helping other people in my community 


i □ 


2 □ 


3 □ 


4 □ 


g. Getting a good education 


i □ 


2 □ 


3 □ 


4 □ 


h. Getting a good job 


i □ 


2 □ 


3 □ 


4 □ 



C2. As things stand now, I think I will . . . 

MARK ONE ONLY 

1 □ Complete high school and graduate with a diploma 

2 □ Drop out of high school and complete the GED 

3 □ Not finish high school 

C3. How far would you like to get in school? 

MARK ONE ONLY 

1 □ High school graduate/GED 

2 □ Technical or trade school 

3 □ Associates degree (2 year college degree) 

4 □ Bachelors degree (4 year college degree) 

5 □ Masters degree or equivalent 

6 □ Ph.D., MD or other advanced degree (like a medical or law degree) 

C4. How likely is it that you will get this far in school? 

MARK ONE ONLY 

Not Very Likely < > Very Likely 

1 □ 2 □ 3 □ 4 □ 5 □ 

0 % - 20 % 21 % - 40 % 41 % - 60 % 61 % - 80 % 81 % - 100 % 
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C5. What reasons might keep you from achieving your educational goals? 

MARK ONE FOR EACH ROW 





Yes 


No 


a. 1 don’t like school 


i □ 


o □ 


b. My grades aren’t high enough 


i □ 


0 □ 


c. Courses are too difficult for me 


i □ 


0 □ 


d. 1 can’t afford it 


i □ 


0 □ 


e. 1 plan to join the military 


i □ 


0 □ 


f. No one in my family has ever gone on to school after high school 


i □ 


0 □ 


g. I’d rather work and make money than go to school 


i □ 


0 □ 


h. 1 don’t think that going to school is important 


i □ 


0 □ 


i. 1 need to help support my family 


i □ 


0 □ 


i. Some other reason (name this reason) 


i □ 


0 □ 



C6. Tell us a little about your career goals. In the boxes below, name up to three careers you would most like 
to have and answer the three related questions about each career. 

MARK ONE FOR EACH ROW 

Name the career or careers 



you would most like to have: Answer these related questions: 


Not Very 




> 


Very 


a. How interested are you in this career? 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 


1 . b. How likelv are vou to enter this career? 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 


c. How well would you perform in this career?.... 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 




a. How interested are you in this career? 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 


2. b. How likelv are vou to enter this career? 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 


c. How well would you perform in this career?.... 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 






a. How interested are you in this career? 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 


3. b. How likelv are vou to enter this career? 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 


c. How well would you perform in this career?.... 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 
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and answer the three related questions. 



C7.r- Name the career you expect to be working in by age 30 



T 



Answer these related questions: 



Not Very <- 



MARK ONE FOR EACH ROW 



-> Very 



a. 


How interested are you in this career? 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 


b. 


How likely are you to enter this career? 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 


c. 


How well would you perform in this career? ... 


i □ 


2 □ 


3 □ 


4 □ 


5 □ 



C8. What education or training do you need for this career? 

MARK ONE ONLY 

1 □ No education after high school is needed 

2 □ Military training 

3 □ Technical or trade school 

4 □ Associates degree (2 year college degree) 

5 □ Bachelors degree (4 year college degree) 

6 □ Masters degree or equivalent 

7 □ Ph.D., MD or other advanced degree (like a medical or law degree) 

8 □ Other ( Please describe) 

d □ Don’t know 



C9. How likely is it that you could successfully complete the education and/or training required to enter this 
career? 




CIO. What reasons might keep you from achieving your career goals by age 30 ? 

MARK ALL THAT APPLY 

1 □ Not enough education 

2 □ Need to work to support my family 

3 □ My parent or guardian wants me to have a different career 

4 □ Other ( Please describe) 

5 □ I can’t think of a reason that will keep me from achieving my career goals 
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Cl 1 . How much do you agree or disagree with each of the following statements? 



MARK ONE FOR EACH ROW 





Strongly 

Disagree 


Disagree 


Agree 


Strongly 

Agree 


a. 1 know what is required to succeed in different careers 


i □ 


2 □ 


3 □ 


4 □ 


b. 1 know how to find out about what types of jobs are best for me 


i □ 


2 □ 


3 □ 


4 □ 


c. 1 have a good idea about the kinds of jobs 1 would be good at 


i □ 


2 □ 


3 □ 


4 □ 


d. 1 will be able to overcome barriers that stand in the way of 

achieving my career goals 


i □ 


2 □ 


3 □ 


4 □ 
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D. BACKGROUND INFORMATION 



D1 . When were you born? 

I_l_l / l_l_l / IJ_I_9_I I I 

Month Day Year 

D2. Are you: 

1 □ Male? 

2 □ Female? 

D3. How do you describe yourself? 

MARK ALL THAT APPLY 

1 □ White 

2 □ Black or African-American 

3 □ Hispanic or Latino/Latina 

4 □ Asian 

5 □ Native Hawaiian or Other Pacific Islander 

6 □ American Indian or Alaskan Native 

7 □ Other (Please describe) 

D4. What is the highest level of education completed by your mother or female guardian? 

MARK ONE ONLY 

1 □ Some high school 

2 □ GED 

3 □ High school graduate 

4 □ Technical or trade school 

5 □ Associates degree (2 year college degree) 

6 □ Bachelors degree (4 year college degree) 

7 □ Masters degree or equivalent 

8 □ Ph.D., MD or other advanced degree (like a medical or law degree) 

9 □ Other (Please describe) 

d □ Don’t know 
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D5. What is the highest level of education completed by your father or male guardian? 

MARK ONE ONLY 

1 □ Some high school 

2 □ GED 

3 □ High school graduate 

4 □ Technical or trade school 

5 □ Associates degree (2 year college degree) 

6 □ Bachelors degree (4 year college degree) 

7 □ Masters degree or equivalent 

8 □ Ph.D., MD or other advanced degree (like a medical or law degree) 

9 □ Other (Please describe) 

d □ Don’t know 

D6. What is the main language spoken at home? 

MARK ONE ONLY 

1 □ English 

2 □ Spanish 

3 □ English and Spanish equally 

4 □ Other (Please describe) 

D7. Do you have a computer at home with access to the internet? 

i □ Yes 

0 □ No 

D8. Please fill in today’s date. 

I_l_l / l_l_l / l_ 2 J_QJ_ 0 J_ 9 J 
Month Day Year 

D9. Do you have any siblings or anyone else in your home who currently attend the 7th grade at this school? 

1 □ Yes 

0 □ No 

DIO. Have you participated in a program or class called Roads to Success? 

1 □ Yes 
o □ No 

Thank you very much for completing this survey. 
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APPENDIX B 

CODING SCHEME FOR REASONABLY AMBITIOUS AND 
REALISTIC CAREER EXPECTATIONS 




We coded the open-ended occupation data from the baseline and follow-up student surveys 
to reflect the degree to which each career was reasonably ambitious and realistic. Two survey 
associates were trained to use the coding scheme and independently coded a small subset of 
occupational data. The codes were then compared to those of an expert coder on the project. 
Discrepancies were noted and resolved through discussion and selection of the correct code. 

Following the initial training, we measured coder reliability and accuracy. Each coder 
independently coded twenty percent of the occupation data, and inter-rater reliability was 
calculated using a Kappa coefficient. The Kappa statistic is a standard approach used to assess 
inter-rater reliability; this approach adjusts for chance agreement between two coders and 
therefore is preferable to assessing raw inter-rater agreement (Landis and Koch 1977). Coders 
attained a Kappa coefficient of 0.86 (98 percent agreement) for the reasonably ambitious/realistic 
expectation scale. Thus, the coding procedure used in this study was deemed reliable across 
coders. Routine checks were performed throughout the coding process to ensure consistency of 
coding and to check for possible “drifting” from the original intent of the coding scheme. 

Coding Scheme 

0 = Meaningless career (responses that are not careers, such as “get good grades” or “make lots of 
money”) 

1 = Very low expectation (for example, dishwasher, fast food, cashier) 

2 = Homemaker (for example, stay-at-home mom/dad) 

3 = Reasonably ambitious/realistic expectation (for example, plumber, doctor, teacher, lawyer) 



4 = Very high/unrealistic expectation (for example, President, professional sports player, model, actress) 




APPENDIX C 

FUTURE OUTCOME VARIABLES 




Although RTS was originally designed as a long-term intervention, beginning in grade 7 and 
continuing through grade 12, this study ended after students in the treatment group had received 
only two years of RTS. Thus this study does not enable us to estimate the full impact of the RTS 
intervention when implemented in grades 7-12. Here we describe outcomes that could be 
measured to estimate effects of the full program, if students were to receive all six years of the 
program. 

Grade 9 Confirmatory Outcome. On “on-track” indicator, previously described by the 
Chicago Consortium (Allensworth and Easton 2005), could be used as a confirmatory outcome 
for grade 9. The Chicago Consortium has used this on-track indicator as an intermediate 
indicator of school performance and has found that it is highly predictive of whether students 
eventually graduate from high school. Among students entering a high school in the Chicago 
Public School District in 1999, those who were on-track by the end of grade 9 were about three 
and one-half times more likely to graduate in four years than were off-track students 
(Allensworth and Easton 2005). Consistent with the Chicago Consortium’s research, we would 
consider a student as on-track at the end of grade 9 if both of the following criteria were met: (1) 
the student has accumulated the number of credits needed to be promoted to grade 10 according 
to district policy, and (2) the student has no more than one semester F (that is, one-half of a full 
credit) in a core subject (English, math, science, or social studies). 

Grade 9 Exploratory Outcomes. Additional exploratory analyses of other outcomes in 
grade 9, including (1) all of those listed for grade 8 (confirmatory and exploratory), (2) grade 9 
grade-point average in core courses, and (3) number of accumulated credits in grade 9, could 
also be investigated. 

Grade 10 Confirmatory Outcome. The confirmatory outcome for grade 10 could be a 
weighted average of the number of credits students have completed with a grade of C or better 
by course type (for example, algebra, other college preparatory work, and vocational education). 
We could weight the types of credits by using their estimated impacts on later outcomes, such as 
college completion or the log of earnings based on data from the National Educational 
Longitudinal Study (NELS) for students with characteristics similar to those of students in the 
RTS program. This measure would incorporate information about courses taken and student 
performance in those courses and should be affected by increases in student engagement in 
school caused by RTS. 

Although some of the literature suggests that the impacts of course taking on test scores are 
ambiguous (Teitelbaum 2003), most research suggests that more rigorous courses have positive 
impacts on test scores (Gamoran and Hannigan 2000), years of education (Allensworth and 
Easton 2005), and later career success (Chaplin 1998) compared to less rigorous courses. The 
confirmatory grade 10 outcome also incorporates course performance that affects grade 
promotion, which in turn is highly correlated with later measures of success (Roderick and 
Nagaoka 2005; Hong and Raudenbush 2005; Allensworth and Easton 2005). 

Grade 10 Exploratory Outcomes. We could also conduct exploratory analyses of other 
outcomes in grade 10, including (1) all of those listed for grade 8 (confirmatory and exploratory); 
(2) the percentages of students taking the SAT, ACT, PSAT, or Pre-ACT in schools where these 
tests are not mandatory; and (3) scores on standardized tests such as those taken for school 
accountability purposes. 




Grade 12 Confirmatory Outcome. We could also use the point system described below to 
measure the degree to which students make progress toward a successful career by the end of 
grade 12. 

For students on track to graduate on time (within six years after entering grade 7) and with a 
regular high school degree, the point system is as follows: 



1. Accepted to college or secured a job that is reasonably ambitious given the student’s 
preparation 1 

2. Applied to at least one college or job that is both reasonably ambitious and 
realistically attainable 2 

3. Applied to at least one college or job, but plans are either not ambitious enough or not 
realistically attainable 

4. On track to graduate but not in the first three categories 



For students not on track to graduate on time with a regular degree: 



5. Still attending high school 

6. Dropped out but have a GED 

7. Dropped out and have no GED 



This grade 12 outcome measure incorporates completed years of education, a factor that is 
highly correlated with later labor market success. In addition, it incorporates information on 
career planning, an important focus of the RTS program. In particular, the “reasonably 
ambitious” and “realistically attainable” caveats for outcomes 1 and 2 help capture the fact that 
RTS is designed to help students align their career and education plans and preparation. 



1 A reasonably ambitious job would be defined as one with earnings in the upper three quartiles at age 28 in 
data from NELS for students with similar grade 12 characteristics (grades and test scores). Thus, for example, 
starting work as an apprentice to an electrician, plumber, mechanic, or carpenter would likely qualify for a student 
with average characteristics. A reasonably ambitious college would be one in the upper three quartiles of what 
students with similar characteristics attended in NELS based on one of the standard measures of college quality 
(such as average freshman SAT scores). Students with below-average test scores would likely satisfy this by 
obtaining admission to any college, including one that is not selective (that is. does not require SAT or ACT scores 
for admission). 

2 A “realistically attainable” college/job would be based on whether the student is in the top three quartiles 
based on their grades and test scores compared to students entering that college/job category in NELS. Thus even if 
a student is rejected by all colleges and jobs, their applications could still count as realistically attainable. 




The impacts of RTS on this outcome may not be linear. To allow for this possibility, we 
could also estimate an ordered logit model and test for the possibility that the impacts of RTS 
vary depending on the level of the variable considered. 

Grade 12 Exploratory Outcomes. The exploratory outcomes for grade 12 could include all 
of the variables listed above for grades 8 and 10. In addition, we could analyze the fraction of 
students who fill out financial aid forms for college (if not required to do so by their high school) 
and analyze postsecondary enrollment using data from the National Student Clearinghouse. 




APPENDIX D 

FULL REGRESSION RESULTS FOR CONFIRMATORY OUTCOMES 




TABLE D.l 



COEFFICIENTS ON CONTROL VARIABLES FROM ANALYSIS OF GRADE 8 CONFIRMATORY 
OUTCOMES (TABLE V.3, LAST COLUMN, p- VALUES) 





Motivation to Go to 
School to Learn Job 
Skills 


Learning and 
Study Habits 
and Preparation 


School Absences and 
Negative Behaviors 




-0.083 


-0.046 


-0.064* 


Roads to Success participation 


(0.267) 


(0.524) 


(0.087) 




0.067 


0.082* 


-0.052* 


Baseline math z-score 


(0.113) 


(0.056) 


(0.073) 




0.028 


0.153*** 


-0.068** 


Baseline reading z-score 


(0.447) 


(0.003) 


(0.016) 




0.225 


-0.251 


0.076 


Black/ African American 


(0.258) 


(0.206) 


(0.670) 




-0.263 


-0.784** 


0.069 


Hispanic 


(0.318) 


(0.029) 


(0.731) 




0.159 


0.317 


-0.096 


Asian 


(0.662) 


(0.343) 


(0.267) 




0.280 


-0.121 


-0.235* 


American Indian 


(0.397) 


(0.745) 


(0.064) 




-0.004 


-0.053 


0.038 


Other race 


(0.964) 


(0.516) 


(0.570) 




-0.080 


-0.189*** 


0.176*** 


Male 


(0.169) 


(0.000) 


(0.000) 




-0.002 


-0.002 


-0.060** 


Mom has at least HS diploma 


(0.978) 


(0.966) 


(0.048) 


Mom has at least bachelor’s 


0.003 


0.024 


0.023 


degree 


(0.971) 


(0.655) 


(0.566) 




0.088 


0.082* 


0.005 


Dad has at least HS diploma 


(0.143) 


(0.069) 


(0.865) 


Dad has at least bachelor's 


-0.015 


0.004 


-0.074 


degree 


(0.804) 


(0.944) 


(0.138) 




-0.061 


0.006 


0.003 


Computer access at home 


(0.494) 


(0.927) 


(0.960) 




-0.492*** 


-0.429* 


0.135 


Non-English spoken at home 


(0.009) 


(0.069) 


(0.423) 




-0.203 


0.025 


0.050 


Over age for grade 


(0.175) 


(0.896) 


(0.717) 


School average free or reduced- 


0.511 


-0.213 


-0.356 


price lunch eligibility 


(0.312) 


(0.673) 


(0.349) 




School average baseline math z- 


-0.041 


-0.068 


0.014 


score 


(0.899) 


(0.837) 


(0.934) 


School average baseline reading 


0.353 


-0.014 


-0.043 


z-score 


(0.233) 


(0.953) 


(0.740) 


Number of observations 


1,287 


1,299 


1,298 



Source: Mathematica student surveys administered in fall 2007 and spring 2009. 

Notes: The control variables include grade 6 test scores by subject (math and reading); race/ethnicity indicators 

(Black/African American, Hispanic, Asian, American Indian, other race); gender; age; parent education 
separately for the mother and father (has at least a high school diploma, has at least a bachelor’s degree); 
primary language at home is not English; internet access at home; school average grade 6 test scores by 
subject; and school average free lunch. The standard deviations of the outcomes are 0.75, 0.73, and 0.55 
respectively. 

* Significantly different from 0 at the .10 level, two-tailed test. 

**Significantly different from 0 at the .05 level, two-tailed test. 

***Significantly different from 0 at the .01 level, two-tailed test. 




APPENDIX E 
SUBGROUP RESULTS 




TABLE E.l 



ESTIMATED SUBGROUP IMPACTS ON GRADE 8 CONFIRMATORY OUTCOMES (TABLE V.3, LAST 

COLUMN, p- VALUES) 





Motivation to Go to 
School to Learn Job 
Skills 


Learning and Study 
Habits and 
Preparation 


School Absences 
and Negative 
Behaviors 


Half of sample with highest 
estimated impacts on career 


0.014 


0.004 


-0.041 


exploration with school staff 


(0.882) 


(0.969) 


(0.353) 


Half of sample with lowest 
estimated impacts on career 


-0.176** 


-0.110 


-0.082** 


exploration with school staff 


(0.019) 


(0.111) 


(0.047) 


Lowest quartile of baseline math 


0.046 


0.071 


-0.139* 


scores 


(0.680) 


(0.488) 


(0.068) 


2nd lowest quartile of baseline 


-0.314*** 


-0.214*** 


-0.019 


math scores 


(0.004) 


(0.008) 


(0.782) 


2nd highest quartile of baseline 


-0.012 


0.021 


-0.115** 


math scores 


(0.908) 


(0.850) 


(0.037) 


Highest quartile of baseline math 


-0.013 


-0.008 


-0.024 


scores 


(0.907) 


(0.943) 


(0.707) 


Lowest quartile of baseline 


-0.030 


-0.019 


-0.090* 


reading scores 


(0.738) 


(0.876) 


(0.099) 


2nd lowest quartile of baseline 


-0.004 


0.026 


-0.021 


reading scores 


(0.977) 


(0.789) 


(0.711) 


2nd highest quartile of baseline 


-0.103 


-0.139 


-0.134** 


reading scores 


(0.259) 


(0.245) 


(0.029) 


Highest quartile of baseline 


-0.193** 


-0.048 


0.017 


reading scores 


(0.030) 


(0.526) 


(0.793) 




-0.067 


-0.026 


-0.024 


Mother has less than HS diploma 


(0.522) 


(0.787) 


(0.635) 


Mother has HS diploma but not a 


- 0.111 


-0.043 


-0.111** 


bachelor’ s degree 


(0.098) 


(0.567) 


(0.047) 




0.063 


-0.036 


-0.018 


Mother has a bachelor’ s degree 


(0.707) 


(0.709) 


(0.865) 




-0.115* 


-0.033 


0.039 


Father has less than HS diploma 


(0.094) 


(0.643) 


(0.410) 


Father has HS diploma but not a 


-0.048 


-0.046 


-0.113** 


bachelor' s degree 


(0.631) 


(0.562) 


(0.021) 




-0.079 


-0.129 


-0.197** 


Father has a bachelor’s degree 


(0.500) 


(0.208) 


(0.029) 




-0.045 


-0.022 


-0.090* 


Male 


(0.652) 


(0.805) 


(0.068) 




